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ABSTRACT 



The concern of this booklet is with "two-way 
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Distance Education — An Empirical Study of the Impact of Various 
Frequencies of Assignment Submission" (Holmberg, Schuemer); *nd 
"Submission Density, Amount of Submission Questions, and Quality of . 
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Introduction 

Bfrje Holmberg, FornUnlversMt/ZlFF 

Whfle distance education can be - and far from seldom is - combined with face-to-face elements, its basic 
characteristic is non-contiguous, I.e. mediated communication. There is usually both one-way traffic in the 
form of pre-produced courses (printed, recorded and/or broadcast) and two-way traffic. It Is with the latter 
that this booklet is concerned. 

Traditionary real communication, i.e. two-way traffic occurs by cottespondence, le. in writing. Computer- 
medteted communication and telephone interaction are other types of communication which are gaining 
increasing importance. Considerable potential is to be ascribed to the possibilities that electronic mail offers 
forwideJayed communication and that computer conferencing makes available for an asynchronous 
exchange of views and experiences. Nevertheless, even in the last decade of the twentieth century written 
communication is likely to dominate distance education. Today's preponderance of written communication is 
weU known. A recent international study showed that 145 out of 171 dlstance4eaching institutions rely on 
"written correspondence* (Sehuemer 1988 p. 55). This written communication is largely based on assignments 
causing students to submit their problem solutions, answers and essays to the tutors of the supporting 
distance-teaching organisation for correction and comment 

This type of written communication has been made the object of some important studies with a view to 
clarifying its functions, methods, frequency and speed. Two already classical works stand out, John Baath's 
monograph of 1980 on postal two-way communication and Torstein Rekkedal's 1973 study of the effects of 
reducing the turn-around times of submission assignments. 

Later studies by Graver DiehJ and the present author In co-operation with Rudolf Sehuemer have replicated 
and/or looked further into the questions analysed by Baath and ReKkedal. in this booklet these later studies 
are reproduced together with comments by Baath and Rekkedai. 

What the Impact of frequent communication and short turn-round times is has not yet been made fully clear. 
Whfle in Rekkedal's study the value of short turn-round times was shown to be statistically significant, DiehJ for 
one (in the first of his papers published here) has come to a different conclusion (cf. also Barker et al 1986) 
Neither BMth nor Holmberg & Sehuemer in the studies referred to above have been able to find statistically 
significant support for their hypotheses about the favourable influence of frequent communication. 

As shown by Baath, DiehJ and Rekkedai in this volume other circumstances than communication frequency 
and speed not only influence the outcome of distance study but can function as intervening variables In 
empirical studies of the type discussed and presented here. Diehl refers to feedback modalities, Baath rightly 
points out that the character and quality of tutor-student interaction most probably influence the results and 
Implies that high communication frequency is likely to be of Importance if the interaction Is of great value to 
students, and Rekkedai explicitly states that feedback interval is only one element and not necessarily the 
most significant one*. 

The relevance of the character and quality of mediated tutoring is evidently great. The report given by 
Sehuemer and me fiats to take this Into account A separate report was planned, but has not yet been written. 
In our experiment the tutors aimed at personal comments on students' individual achievements, avoided 
merely ticking off what was correct or less correct in favour of personal notes, but a! the same time used pre- 
produced comments on expected difficulties; experiences of the latter practice has in other contexts been 
favourable (Rekkedai & Ljosa 1974). In a later study pre-produced comments were developed further and 
used in a PC supported course model (Fritsch 1989, Wute 1989 and Kuffner 1989). Whether this type of tutorial 
support meets Baath's requirement that it should be 'of great value to the students' remains a problem. 

When the possible value of short turn-round times of students* assignments is judged It is no doubt important 
on the one hand to consider the diminishing need of prompt mediated (written) feedback In the cases when 
supplementary face-to-face tuition is also provided, which is relevant in the Barker et al. study, on the other 
hand to make dear what is meant by short turn-round times (In Rekkedal's study up to a week) and delayed 
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feedback. It Is Hunting to see what the much quoted KuJhavy has hi mind; he himself says that his studies 
show that 'people remember correct ftnswers just as well when feedback Is delayed a day as when it Is given 
Immediately after the response' (Kulhavy 1977 p 214). 

I personally stffl assume that, given the right circumstances, both hfch communication frequency and short 
turn-round times can favour students' motivation and achievements, but there Is no denying that there Is room 
for hesitation and uncertainty. The well-considered comments by Baath and Rekkedal to Dtehl's and Holmberg 
& Scbuerner's new contributions to the debate fflumlnate the situation. 

The uncertainty in these cases Is nothing unusual or unexpected. Education is concerned with human beings 
with personalities, hopes, and wills of their own. If we are not determlnlsts In the sense that we totally reject the 
assumption that human wffl Is in any respect free, then it is Impossible to postulate any automatic cause-effect 
principle in research that aims at optimizing educational methods and procedures. Here theories usually have 
to be limited to statements to the effect that & such and such a measure is taken under specific circumstances, 
then this is likely to facilitate learning. 

It Is, as expressed by Hosford, 'Impossible to determine an absolute set of Instructional procedures that wfll be 
"best", for different learners, or for dWerent learnings by one learner' (Hosford 1973 p. 114). 
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Some Thoughts on Delayed and Immediate Feedback 

December 1982 

Grover E. Dlehl, 
The USAF Extension Course institute, 
GunterAJr Force Base, Alabama, USA 

Popular opinion, even In the education community Itself, Is currently perpetuating a serious 
misconception concerning the question of delayed versus Immediate feedback of examination 
results. 

It Is widely believed In education circles that the immediate feedback (IF) of test results is 
instrumental In increasing student learning Not only, however, are the effects of IF not as 
generally Imagined, the opposite is often true, Delaying feedback for a day or two improves retest 
performance in those situations paralleling most adult Instructional scenarios. This phenomenon, 
known as the Delay-Retention Effect (DRE), Is well documented In the literature (see Blehler and 
Snowman, 1982; Kuihavy, 1977; Surber and Anderson, 1975; and Kulhavy and Anderson, 1972; 
among others) but Is little known outside of the testing community. This lack of widespread 
recognition is unfortunate since DRE can be a valuable asset in learning Improvement. 

Feedback v ersus Reinforcement 

Pei Taps the basic problem In the popularization of DRE is that it requires a change In what 
has been a standard instructional outlook. IF gained widespread currency with the proliferation of 
programmed instruction in the late fifties and sixties. This technique relied heavily on behavioral 
conditioning theory developed by B.F. Skinner at Harvard. The early successes of "programmed 
learning" in small-scale instructional sequences, coupled with a basic confusion of the terms 
feedback and reinforcement, have led to the present uncritical acceptance of IF as an unalloyed 
virtue of instructional technique. 

Many educators continue to equate immediate feedback with behavioral conditioning's 
immediate reinforcement, correctly observing the both immediately follow a response made by 
an individual. Unfortunately they miss two important points. First, "reinforcement 0 is defined as 
some "thing" following a response which measurably alters the probability of that response 
recurring. Feedback has been dropped into the reinforcement category without experimental 
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confirmation that ft performs as imagined; In fact it does not. Second, educators have mistakenly 
applied the behavioral conditioning model (with Its emphasis on simple skill learning among 
rodents, birds, and small children) to the real world of complex learning tasks required of most 
adults. Behavioral theory is very straightforward, but works well only with small units of 
Instruction. The long term retention problems of general education require more comprehensive 
solutions than the behavioral or Skinnenan model offers. 

Whv Delaying Feedback Works 

The Delay-Retention Effect (DRE) is best explained using the "interference-perseveration 
theory" developed by KuJhavy and Anderson (1972). Basically, over the passage of time correct 
responses are remembered whHe Incorrect responses are forgotten. The immediate Interjection 
of feedback following examination, however, interrupts the remembering-forgetting process 
(catted proactive Interference). Thus, it is more efficient to simply refrain from interfering with the 
remembering of the correct responses on a test and le. ti anew the smaller group of previously 
Incorrect questions at a later time. 

In substance of course DRE is considerably more involved than the above explanation 
would suggest. The main point Is that DRE delivers a different set of instructional constructs tnan 
we are accustomed to receiving from behavioral conditioning theory. Essentially, the more 
complex the task or question, the less the behavioral model applies and the more DRE becomes 
a dominant Instructional mechanism. 



implementing PRE 

Standardized instructions for employing DRE are not available since the requirements of 
each educational enterprise are highly Individualized. It is really up to each school to determine 
which mix of immediate and delayed feedback works best for them. It should be noted that 
pro-DRE findings reported in the literature and upon which this essay is based are derived from 
resident school environments and have delays of a day or two. This may contr. -.t with some 
extension education formats and may affect the generallzabiltty of the findings. Now some 
specifics. 

Should we return the answers to all of the questions, or just the ones the student missed? 
The literature discussing the value of correct versus Incorrect answer feedback is a mixed bag. 
Research using normal educational settings (and supporting DP.E) Indicates that knowledge of 
incorrect responses is more "frstructlonaT than knowledge of correct responses (which have no 
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reward or reinforcement value and are simply passed over). Other research (In support of IF) 
contrasts wtth this, finding that knowledge of correct responses is the dominant operant. The 
tetter experiments, however, use highly specialized teaming tasks not commonly considered 
educational. Thus, In general education It seems that feedback of Incorrect responses is 
beneficial, while feedback of correct responses Is for the most part irrelevant. 

Next, feedback must be made available after the response If student learning Is to be 
enhanced. If feedback (in the form of answers to test questions) is available either before or 
during tho examination, students tend to cheat For extr x t>«iy simple tasks cheating is not 
necessarily counterproductive; In complex tasks, however, cheating should be strongly 
discouraged or avoided altogether by procedural barriers. 

There are also two instructional components which strongly affect the contribution of 
feedback to the learning situation. The first is that DRE will be effective only to the degree that the 
student cannot short-circuit the study-examination process. Obviously cheating is a factor here. 
M-rfe important instructtonally, though, If students are permitted to answer correctly due to cues 
or some other external mechanism rather than through an understanding of the material, they win 
tend to simply read the examination and receive little learning enhancement. The second factor 
DRE wfll be more or less effective depending upon the degree to which students understand the 
material and have confidence In their answers. If students must guess and rely on chance alone 
for correct answers, the use of DRE as an Instructional tool becomes Irrelevant and, in tact, IF 
becomes the more powerful variable. The extreme of this condition Is one of no unterstandlng 
and the student teaming that "A" is correct for Item 1 , "D" for Item 2, and so on. 

As It applies to extension education then, the literature on DRE may be summed Into the 
following points: 

t.) Ensure that students can understand the material and have confidence in their answers given 
appropriate preparation. 

2 ) For complex tasks and questions, do not allow students to short-circuit the (earning - 
examination process, or the cheat. 

3. ) For very simple tasks use the immediate feedback of all responses. 

4. ) As tasks become more complex, delaying feedback and providing onlv answers to incorrect 

responses assumes distinct Instructional advantages. 
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An Eamate of * Mixed Eaatea syr^m 

In the preparation of texts and examinations the Extension Course Institute (EC1) uses a 
systems approach (the Irtstructlonal Systems Development (ISO) model), selecting Instructional 
strategies appropriate to the process underway. In ISO students are led progressively, step by 
step, from small, discrete units of Instruction (much as In behavioral conditioning) to the fuU 
understanding of complex processes and concepts 

Each volume of an EO course is composed of a number of short learning objective 
segments, each with a set of test questions. The questions are numerous, come in a variety of 
types, and generally exhaust the content of the objective. Feedback is Immediate since answers 
are provided in another section of the text Upon completion of the volume the student takes a 
Volume Review Extercise (VRE), a multiple choice examination sampling the content of the text. 
Answers are not provided; but since the test is open-book students have "conditional 0 feedback. 
The VREe are scored at ECI and students Informed as to which questions were missed. Correct 
answers are not provided, and students must research the text to Jeam the answers to the 
questions they missed. The final step is a closed-book end-of-course examination sampling ail 
the texts In a course. Here the studb.it is Informed of pass or fail, percentage score, and which 
learning objective segments In the texts relate to the questions missed. 

Summary 

Since this discussion Is necessarily brief, readers are strongly encouraged to individually 
peruse the literature (an excellent bibliography is found In Kulhavy (1977), discuss the subject 
with colleagues, and draw their own conclusions. Two points, however, should be particularly 
ralevant to extension educators. One is that the delays coincident with the process we employ 
have in fact worthwhile educational advantages. The other is that "conventional wisdom" (in the 
present case "assuming* the effectiveness of immediate feedback) is not always correct and must 
be verified. 
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The Effect of Field Scoring on Time To Completion in 
Career Development Courses 

February 1989 
Grover E. D'zni 
USAF Extension Course Institute 
GunterAir Force Base, Alabama, USA 

For many years prior to 1988 the Extension Course Institute (ECI) used Intermediate formative 
examinations, known as Volume Review Exercises (VRE), as an Integral phase of Instruction. After completing 
each volume of a course, the student completed an open book multiple choice test by marking the answers 
on a machine scombJe answer sheet. The answer sheet was then sent to ECI for scoring. Elapsed time from 
the student taking the exam to student receiving feedback from ECI was generally under two weeks. Central 
scoring of the examinations provided an opportunity to collect performance data on the Individual volumes. 
There was also opinion that Institutional scoring provided an affective link between the student and the 
organization preparing the instruction. 

In the late 1980s opinion regarding the VREs shifted causing a reappraisal of the service. (Simultaneously, 
enormous changes were being made in the entire Career Development Course (CDC) program including a 
new instructional format and automated course development.) Many, but not all. of those Involved In test 
development observed that the data provided by the VRE Item analyses and summary statistics were seldom 
used, and argued that cutting the VREs would generate a cost saving. The Idea was not new, but cast Into the 
light of the new times It appeared more reasonable. A so, the preaddressed VRE answer sheets had been 
used as de facto mailing labels for sending courses to students. Operations personnel observed that 
eliminating both envelope stuffing and use of expensive double sided tape (Instead using standard gummed 
labels) could expedite shipping and cost less. There was also some opinion that student/supervisor contact 
was perhaps more Important than student/ECi contact. If the VREs were scored in the field, the supervisor 
would have an opportunity to Interact directly with the student Actual performance outcomes of this notion 
were, however, not clearly enunciated. Finally, there was a popular notion that immediate feedback, via field 
scoring, was superior to institutional scoring, evidence to the contrary notwithstanding (Diehl 1982). 

Regardless of cause, In January 1988 ECI Inaugurated field scoring. Using a phased approach, all courses 
contained field scorable materials by summer of the same year. The first set of fully field scorable courses, 
numbering 16, was activated on 4 January 1988. As part of the conversion process, the Commandant ECi, 
directed that the Evaluation and Research Branch conduct a comprehensive examination of the new approach 
at the first opportunity and provide a report of he findings. The two bask: questions to be asked were: does 
field scoring affect completion schedules, and does field scoring affect student performance (as measured by 
end -of -course test scores). The present study addresses the first question. 
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Subjects . There were two groups of students enrolled in Career Development Courses selected for this 
study. The first consisted of ail finishing students enrolling in CDCs In January 1985, 1986, 1987, and 1988. 
The second group consisted of those completing courses (pass or fail) In November of the same years. For 
January enrollments, only students completing their courses in under 395 days were selected; for November 
completions, only those with under 335 days to completion were taken. Under normal circumstances 
students have as long as 18 months (with extensions) to complete CDCs. A correction was necessary in this 
Instance since the cutoff date for the 1988 sample was considerably less than the maximum available to 
students generally. It can be demonstrated that over 97 percent of the students, in the courses selected, 
normally complete their studies In 12 months or less and the sampling periods were considered acceptable 
compromises to ensure both comparability and timeliness of the data. 

Analysis Plan . The basic research question - has field scoring made any difference in time to completion? 
- was addressed with the following specific tasks: 

(1 ) What Is the average time to completion for students who began their courses in January? 
Is any difference significant? 

(2) How long did students who began their courses in January take to complete each volume? 
Is any difference significant? 

(3) What Is the average time to completion for students who completed their courses In November? 
Is any difference significant? 

(4) How long did students who completed their courses In November take to complete each volume? 
Is any difference significant? 

Descriptive data and Analysis of Covariance tables were prepared for each task. Analysis of Covariance 
was selected as the Inferential technique since the time to completion is known to be overwhelmingly 
determined by the course and how many volumes it contains. Generally, short courses take less time but 
longer per volume than do long courses. Using course as e covariate and extracting the variance prior to 
examination of the main effect corrected for this and allowed the effect of field scoring to be examined in a 
much cleaner environment Probabilities were reported in increments of < .25 to .10 (< .25), < .10 of .05 (< .10), 
< 05 to .01 (<05) and <.01. Only statistics with probabilities less than one percent (p<.0l) were considered 
significant. 

Given the repetitive nature of the analysis, this had the practical result of an overall significance level of 
approximately five percent 

The following objective definitions were established: 

1.) ENRLTiME « Total enrollment time from registration to course completion by pass or faB. This was a 
dependent variable. 
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2. ) QAYSXVOL « ENRLTIME divided by the number of volumes In the course. This was a dependent 

variable, 

3. ) CDC1 toCDClS - Effect coding of CDC number to convert th8 16 CDCs under study into a set of 

categorical variables suitable for mathematical analysis This became the cov?riate In the study. 

4. ) GROUPS « Effect coding to obtain a dlchotomous variable consisting of FY85 to FY87 enrollments (the 

old Institutional scoring system) and FY88 enrollments (students completing courses under field scoring). 
This was the main effect, or treatment, under consideration In the study. 

Statistical Routines. SPSS routines FREQUENCIES, BREAKDOWN and REGRESSION as described in 
SPSS-X Release 2.2 (SPSS, 1986) and available on the Honeywell 6000 computer serving Maxwell and Gunter 
Air Force Bases were used. COMPUTE statements created effect coded variables and SELECT IF 
STATEMENTS segmented the full data fie into the desired groupings. Analysis of Covariance corresponding 
to that described by Keriinger and Pedhazur (1973) was obtained through REGRESSION using the form 

REGRESSION VARS-DAYSXVOL ENRLTIME CDC1 TO C0C15 GROUPS/ 

0EP-DAYSXV0L ENRLTIME/ 

ENTER CDC1 TO CDC15/ENTER GROUPS/ 

OESCRIPTIVES DEFAULTS SIS/ 
STATISTICS ALL 

Results and Discussion 

January Enrollments 

The suggestion by some observers that field scoring would reduce completion time is not supported by the 
present data. As shown in Table 1, the average time to completion under the old method was about 55 days, 
while under field scoring the time has risen to over 65 days. These data are not fully reliable since the totals 
are unweighted between groups, although they are weighted within groups. The mean of the course 
averages, however, may be considered a validity check, and these are < i the same direction. Here, the FY85- 
87 group averaged about 71 days while the FY88 group averaged over 75. After accounting for CDC taken, 
see Table 2, the true Impact of field scoring is shown to oe very small (less than 1/10 of 1%) and not 
significant (p<.25;. The R2 for CDCs Is .659, Indicating that almost two thirds of the total variance of time to 
complete each volume Is accounted for by the course alone, a finding within the desired significance region. 

The same conclusion holds for total time to completion. As shown In Table 3, after accounting for CDC 
taken, the change to field scoring was associated with a very small difference In completion time (about 1 /1C 
of 1%) and was not significant (p<.10). The R2 of the covariate, however, dropped considerably (almost in 
half) suggesting that taking time per volume (as in Table 2) corrected for the fact that shorter courses tend to 
take longer per volume. 
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Table 1: Mean Days to Completion* of Each Volume within 




COCs for January Enrollments 






CDC FY88-87/N Students 


FY88/N Students 


Hr Vols 


11350B 


42.9167/12 


* 


5 


27132 


53.5904/94 


55. 6071/21 


4 


30451 


41.6878/41 


42.2222/9 


5 


30750 


41.5755/71 


62.3438/24 


4 


32358C 


112.5789/19 


103.8375/40 


2 


32853 


47.6839/143 


53.8500/28 


5 


42350 


58.4970/167 


61.2560/56 


3 


42652 


42.3616/325 


43.3520/107 


6 


42753 


57.4505/37 


58.6111/12 


3 






Note: as of 6 Jun 86: 


4 vols 


43152C 


63.3139/154 


55.3718/52 


3 


46450 


45.5000/30 


51.4444/3 


0 


49152 


54.BS23/26 


53.1389/36 


3 


55131 






3 


61251 


119.2821/39 


125.0000/3 


1 


672738 


223.7000/10 


222.1875/16 


1 


74131 


58.7586/58 


58.7500/16 


^ 


Entire 


55.1940/1226 


65.3009/423 




Population 








Average of 


70.9059 


75.3552 




Means 








* (Total days to completion/number of volumes }/N 



Table 2: Summary of Analysis of Covariance Examining the Effect of Scoring Treatment 




on Average Time to Completion for Volumes within Courses among January 




Enrollments 


: N - 1649 












Source R? 


SS 


df 


MS 


F 


p< 




Covariate .659 


1159551 


14* 


82825 


225.068 


.01 




Treatments .000 


750 


1 


750 


2.038 


.25 




Error ,341 


600286 


1633 


368 








Total 1.0 


1780587 


1648 










* one CDC. 55131, had 


no enrollments du 


ring the period 
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Table 3: Stannary of Analysis of Covartance Examining the Effect of Scoring Treatment 
on Total Time to Completion among January Enrollments: H « 1649 



Source 


R2 


SS 


df 


MS 


F 


J* 


Covariate 


.337 


2163202 


14* 


154514 


59.428 


.01 


Treatments 


.001 


7118 


1 


7118 


2.736 


.10 


Error 


.662 


4246918 


1633 


2600 






Total 


1.0 


6417238 


1648 









* one CDC, 55131, had no enrollments during the period 



November Completions 

Examining data for November completions is a validity check on the January sample, instead of looking 
forward to how long ft took students to complete, the focus is on when students enrolled given a fixed 
completion point Although there is some overlap between the January and November samples, it is only 105 
students out of 2885. These are under four percent and, from examination of frequency data, are spre* nore 
or less randomly among the CDCs. Sampling bias is not apparent 

As with the January data, there Is an apparent shift among the field scored students towards longer 
enrollment times. From Figure 1. ft appears that students completing in November 1988 tended to enroll 
earlier in the year than did comparable students in November 1985, 86 and 87. and the distribution within the 
enrollment period is flatter. An Interesting anomaly in the figure occurs at the NOV entry along the baseline. 
Although t s difficult to see, the raw data show .3 percent of the November 1988 completions actually enrolled 
the same month. For years 1985 through 1987 there were no November cancellations who enrolled the same 
month. Although tnls seems incongruous with the overall observation that field scoring students take longer, it 
actually provides additional support. 



ERIC 
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NOVEMBER COMPLETIONS - 

WHEN STUDENTS ENROLLED 
25-. 




NOV OCT SEP HUC JUL JUN MAY RPR MRR FEB J FIN 

ENROLLMENT MONTH 



1985-87 



Enrollment months for students completing Career Development Courses In the month 
of November In years 1985 through 1988. 
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Prior to 1968 end- course examinations were triggered by return of the last VRE answer sheet, which 
required a turnaround time of about two weeks to get an exam back to the student for administration. Given 
the outset maB transit time of the course packages (normally 2 to 3 weeks (DtehJ 1988 and 1984)), it was 
administratively impossible for a student to complete a course - registration to completion of the exam - in 
under 30 o>ys. Under Beta scoring, end-of-course examinations are forwarded via an automated system at 
the request of the student, eliminating the previous two week turnaround. Thus, it became theoretically 
possible to complete quickly and .3 percent of the students actually took advantage c f tha opportunity. 
Unfortunately, the opportunity did not translate into an across-the-board two-week reduction In completion 
time. 

Examining the data in Table 4 suggests that students completing in November 1988 actually began their 
courses earlier than in previous years, thus taking relatively longer to complete. Grouped, the average length 
of time for each volume in the courses was 58 and 65 days for FY1 985-78 and FY88, respectively. The means 
of the averages were 68 and 76, respectively. Again, however, these apparent differences are not statistical'*/ 
significant The Analysis of Covsdance for total days to completion, the statistical test for the data shown In 
Figure 1, are at Table 5. After accounting for CDC taken, the proportion of time to completion variance 
accounted for by the type of VRE scoring was about .2 percent. The probability of this statistic was .05. which 
was not significant under the parameters of the present study. Even if It had been significant the extremely low 
R2 (.002) is of little more than academic Interest. It is difficult to Imagine what type of management decision 
would be made on a finding this small. Also in Table 5, note that the R2 of the covariate is at about the same 
level as that in Table 4, the equivalent analysis for the January enrollments. 
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1 1 — - — — . 

Table 4: Mean Days to Cccjpl»t1cn* of Each Volume within 




CDC* for November Ctanptet Ions 






CDC FY88-87/H Students 


rioo/n Students 


Nr Vols 


113 SOB 


52 9571/7 




5 


27132 


57 0372/47 


. cco3/c3 


4 


30451 


W . £0011/ £0 


44 ft^Ofl/t? 


5 


30750 


<30 • 3010/ 07 


0/ • //9U/10 


4 


32658C 


AIM/ • VI OA/ PC 


HZ- 5454/31 


2 


32653 


47 M47/114 


to ,4929/ co 


5 


42350 


57.6588/115 


62.7333/45 


3 


42652 


42.1758/281 


40.9297/83 


6 




IftftO/^Q 
79 • *OOV/ 39 


51.U9UD/11 


3 






note: as of 6 Jun 8 


8 * vols 


43152C 


62.8652/136 


56.2473/31 


2> 


46450 


41.1667/8 


42.6333/5 


6 


49152 


52-5532/47 


65. 5789/19 


3 


55131 






3 


61251 


125.5*231/52 


169.7500/8 


1 


672738 


186.2353/17 


177.7857/14 


1 

a 


74131 


59.0612/49 


82.8333/4 


3 


Entire 


58.3171/1024 


65.6115/320 




Population 








Average of 


68.4975 

t 


76.1033 




Means 






* S (Total days to completion/number of volumes)/** 



Table 5: Sunwary of Analysis of Co variance Examining the Effect of Scoring Treatment on Total Time 
to Completion among November Completions: N ■ 1348 



Source 


R2 


ss 


df 


MS 


F 


P< 


Covariate 


.350 


1563349 


14* 


111687 


51.294 


.01 


Treatments 


.002 


8678 


1 


8678 


3.986 


.05 


Error 


.648 


2899690 


1332 


2177 






Total 


1.0 


4471716 


1347 









* one CDC, 55131, had no enrollments during the period 
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Table 6: Suarory of Analysis of Covar a nee Examining the Effect of Scoring Treatment on Average Time 
to Completion for Volumes within Courses among November Completions: H * 1348 



Source 


RZ 


SS 


df 


MS 


F 


P« 


Covar late 


.632 


1085963 


14* 


77570 


164.343 


.01 


Treatments 


.001 


1810 


I 


1810 


3.835 


.10 


Error 


.366 


629196 


1332 


472 






Total 


1.0 


1716989 


1347 









* one CDC. 55131, had no enrollments during the period 



Finally, the Information available on time to completion for each volume for the November completions is 
similar to that for the January enrollments, in Table 6, the effect due to treatments (type of VRE scoring) 
remains smell and not significant, while the covariate (CDC taken) accounts for a significant proportion of the 
variabfl&y In enrollment time (over 63%, p< .01). These data are extremely close to those in Table 2. 

Conclusion 

it is clear that the implementation of field scoring has opened a window for accelerated course completion, 
of which students have overwhelmingly fatted to take advantage. On the basis of both total time to completion 
and time to completion for individual volumes, field scoring has made no impact that Is either statistically or 
practically significant. Time to completion is predominantly a function of the course in which the student is 
enrolled. 

There was In mis study no attempt to either prove or disprove any of the a priori notions initially provided in 
support of the change from institutional to field scoring of the volume review exercises, indeed, to do so 
would have been completely presumptive as there Is no single justification for the change, and the arguments 
differ depending upon the ECI division being interviewed. For this reason, the discussion was intentionally 
non-dlrecttonal and the reader is encouraged to evaluate the data independently. 
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Comparison of Two Pre-test Feedback Modalities 
on End o* Course Test Performance 

March 1989 
GrvverE. Dieht 
USAF Extension Course Institute 
GvnterAir Force Base, Alabama, USA 

The effect of quick feedback on student performance is open to considerable conjecture In the general 
educational community and It depends greatly on the content and method of Instruction. The Immediate 
feedback proponents generally advance a simplistic behavioral conditioning argument and It works weii for 
simple task acquisition. KuJhavy and Anderson (1972) were among the first to seriously question whether there 
might be a positive "Interference-perseveration" effect especially in the more complex cognitive tasks. 
Basteafly, they noted that people tend to forget what they do not know wen and that these are the Items 
generaBy missed on examinations. If there Is a delay in providing feedback, students remember relatively more 
of the correct information; immediate feedback, however, Introduces proactive interference which interrupts 
the forgetting process. 

The present study examines the impact of delayed versus immediate feedback (or turn-around time) from a 
slightly different perspective. Prior to taking the final examination each Extension Course institute (EC!) 
student completes intermediate formative experiences, called volume review exercises (VRE). In appearance 
these are exactly like the final examinations but the VREs are open book while the final exams are proctored, 
cfosed book sumrnative Instruments, Prior to January 1988, students recorded their answers to the VREs on 
separate answer sheets which were maBed to ECl for scoring. Results and Instructional guidance were mailed 
back. The turn-around time for the procedure was about two weeks. For a number of reasons, including a 
desire to reduce the turnaround time and perhaps positively influence student performance, ECl Instituted 
field scoring in January 1988. Via this mechanism there was an opportunity to make student performance 
feedback Immediate. Under both procedures students have Immediate "conditional" Information on their 
performance (the open book aspect). The difference between the two groups Is in the timeliness of formal 
assurance. 



Method 

Subjects. There were two groups of students enrolled In Career Development Courses selected for this 
study. The first consisted of all finishing students who enrolled in CDCs In January 1986, 1986, 1987, and 
198a The second group consisted of those completing courses (pass or fall) in November of the same years. 
For January enrollments, only students completing their courses In under 395 days were selected; for 
November completions, only those with under 335 days to completion were taker.. Under normal 
circumstances students have as long as 18 months (with extensions) to complete CDCs. A correction was 
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necessary in this instance since the cutoff date for the 198P sample was considerably less than the maximum 
available to students generally. It can be demonstrated that over 97 percent of the students In the courses 
selected normally complete their studies In 12 months or less and the sampling periods were considered 
acceptable compromises to ensure both comparability and timeliness of the date. 

Analysis Plan, The basic research question ~ has field scoring of the VREs made any difference in student 
end of course test performance? - was addressed with the following specific tasks: 

1. Under each feedback condition, what is the average score for students who began their courses in 
January? Is any difference significant? 

2. Under each feedback condition, what is the average score for students who completed their courses in 
November? Is any difference significant? 

Descriptive data and Analysis of Covariance tables were prepared for each task. Analysis of Covariance 
was selected as the inferential technique since test score is overwhelmingly dependent upon the course and 
Individual test form Using test form as a covariate and extracting the variance prior to examination of the 
main effect corrected for this ami allowed the enact of field scoring to be examined In a much cleaner 
environment. Probabilities were reported In increments of <.25 to .10 (<.25), <.10 of .05 (<.10). < 05 to .01 
(<.05) and <.01. Only statistics with probabilities less than five percent (p< .05) were considered significant. 

The following objective definitions were established: 

1 .) TESTSCOR « Student score on the final examination and the dependent vas iarte in the study. 
Students falling an Initial end of course examination are offered a retest. Those not taking the retest are 
considered non-completions; those failing the retest are course failures; those passing are successful 
completions. Only the last two categories are included In the present study. 

A note on the psychometric characteristics of the tests. Alt are four option multiple choice examinations 
with a maximum of 124 Herns randomly selected from hundreds, often thousands, available for each 
course. Although the number of Items tends to decrease over time (bad Items are deleted, no new Hems 
are added and there are no substantive corrections), unpublished homogeneity of variance tests 
performed on samples of 51, 126 and 201 students in 1981 demonstrated that the initial randomization 
procedure produces remarkable stability. 

2.) T1 1321 to T74129 ■ A set of 56 effect coded vectors to account for the 57 separate end of course 
examinations encountered in the study. The actual number of tests applicable to January enrollments and 
November completions is less due to selection fluctuations. This was the covariate in the study. 

a) GROUPS - Effect coding to obtain a dichotomous variable consisting of CY85 to CY87 enrollments 
(the dd institutional scoring system) and CY88 enrollments (students completing courses under field 
scoring). This was the main effect, or treatment, under consideration in the study. 
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Statistical Routines . SPSS loutlnes FREQUENCIES, BREAKDOWN and REGRESSION as described in 

SPSS-X Release 2.2 (SPSS, 1986) and available on the Honeywell 6000 computer serving Maxwell and Gutter 

Air Force Bases were used. COMPUTE state *irts created effect coded variables and SELECT IF 

STATEMENTS segmented the full data ffie into the desired groupings. Analysis of Covariance corresponding 

to that dest •'bed by Kerllnppc and Pedhazur (1973) was obtained through REGRESSION using the form 

REGRESSION VARS»TESTSOOR T1 1321 TO T74129 GROUPS/ 
DEP=TESTSCOR/ 

ENTER T1 1321 TO T7412*/ENTER GROUPS/ 
DESCRIPTIVES DEFAULTS SIG/ 
STATISTICS ALL 



Results and Discussion 

January Enrollments 

Despite the large number of examinations and diversity of subject matter in the courses selected for this 
study, the mean scores and standard deviations are ramarlcbty close. As shown in Table 1, the overall 
average mean score was 81.5, with a standard de **tton of 3.7. The field scored courses were slightly higher 
than the Institutionally scored courses, 82.8 versus 81.1, respectively, although the field scored courses had 
relatively more variability (SD for CY85-67 was 3.8 and for CY88 it was 4.4). For the entire 1649 students in the 
sample taken together, the grand mean was 81 .31 and standard deviation was 9.029. 

The data In Table 1 strongly suggest that there is more variability among tests than between the two VRE 
scoring methods. This impression is substantiated In ihe Analysis of Covariance. As shown in Table 2, the 
covariate - test form - accounted for over 1 1 percent of the variability 'n student performance After removing 
this variance, the effect due to treatment - the VRE scoring procedure - added virtually no new variance to 
the model (less than .0005 percent). The F for the covariate had a probability of less than .01. The F for 
treatment, however, did not even requ.'.v calculation as the mean square due to treatment was less than the 
mean square In the error term. In the -January enrollments, then, the implementation of VRE field scoring 
made no significant difference in student performance. 
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Table I: Descriptive Statistics for End of Coarse Examinations among January Enrollments 
during 1965 through 1888: H « 1649 



Enrollment tear 





"test 




19B5-B7 










Total 1985-88 


CDC 


Form 


Bean 


SO 


n 


Mean 




N 








U350 


21 


BO. 1 


7.0 


7 










7 O 


7 




22 


79.5 


12.0 


2 








79.5 


12.0 


2 




23 


84.0 


0.0 


1 








84.0 


0.0 


1 




24 


77.5 


2.1 


2 








77.5 


2.1 


2 




Total: 


Mean 


• 79.9, 


SO » 


6.6, n « 












27132 


33 


77.4 


11.8 


42 








77.4 


11.8 


42 




34 


78.1 


10.0 


38 








76.1 


10.0 


38 




35 


72.3 


13.5 


6 


75.3 


8.4 


10 


74.2 


10.3 


16 




36 


73.9 


16.3 


8 


74.5 


9.7 


n 


74.3 


12.5 


19 



Total: Mean * 76. 0, SD * 11. 1, N « U5 



30451 33 


85.0 


7.5 


21 80.3 9.0 


3 


84.5 


7.7 


24 




34 


85.8 


6.1 


20 84.3 3.3 


6 


85.4 


5.5 


26 




Total: 


Mean *> 


85.0. 


SD * 6.6, n " SO 












30750 25 


78.3 


11.8 






78.3 


11.8 


33 




26 


85.5 


8.9 


38 




85.5 


8.9 


38 




29 






89.7 5.9 


14 


89.7 


5.9 


14 




30 






87.1 6.2 


13 


87.1 


6.2 


10 




Total: 


Mean » 


83.8, 


SD « 10.2, N ■ 95 












32568C 23 


85.3 


9.8 


9 86.8 8.0 




86.4 


8.4 


32 




24 


82.3 


7.6 


10 85.5 5.2 


17 


84.3 


6.3 


27 




Total: 


Hean ■ 


85.4, 


SD - 7.5, H « 59 












32853 23 


79.2 


10.4 


28 




79.2 


10,4 


28 




24 


81.9 


10.2 


17 




81.9 


10.2 


17 




25 


80.4 


8.9 


40 78.2 8.3 


14 


79.9 


8.7 


54 




26 


82.6 


9.4 


58 82.5 9.2 


54 


82.6 


9.3 


72 




Total: 


Hean • 


81.1, 


SD - 9.4, H - 171 













(Continued) 
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Table 1 (continued) 



Descriptive Statistics for End of Course Examinations among .January Enrollments 
during 198S through 1988: H * 1649 



Enrollment Year 

Test 1985-87 1988 Total 1985-88 



CDC Form Mean SO N Mean SD H Mean SO N 



42350 



Z3 


84.7 


d.l 


90 




84.7 


8.1 


90 


24 


84.1 


8.9 


77 




84.1 


8.9 


77 


25 






79.8 5.7 




79.8 


5.7 


29 


26 






80-6 5.6 


27 


80.6 


5.6 


27 


Total: 


Mean ■ 


83-3. 


SO * 8.0, H - 223 











42652 



23 


78.8 


4 5 


4 








78.8 


4.5 


4 


25 


78.5 


7.4 


50 








78.5 


7.4 


50 


26 


77.7 


8.4 


48 








77.7 


8.4 


46 


27 


60.5 


8.1 


97 








8^ 5 


8.1 


97 


28 


80.4 


7.9 


128 








80.0 


7.9 


128 


29 








78.2 


10.7 


52 


78.2 


.0.7 


52 


30 








79.4 


8.7 


55 


79.4 


8.7 


55 



Total: Mean = 79.5. SO = 8.4, N * 432 



42753 



21 
22 



76.0 
78.0 



8.1 
5.9 



22 
15 



80.7 
79.7 



4.7 
11.3 



6 
6 



77.0 
78.5 



7.7 
7.6 



26 

21 



Total: Mean ■ 77.6, SD » 7.u, H * 49 



43152 21 82.6 8.4 81 80.6 8.4 22 82.2 8.4 103 
22 81.4 8.$ 73 84.3 6.2 30 82.3 8.0 103 

Total: Mean « 82.2. SD * 8.2, N « 206 



46450 



32 


84.2 


4.8 


5 










64.2 


4.8 


5 


33 


88.5 


9.2 


2 










88.5 


9.2 


2 


34 


84.6 


8.6 


14 


94 


0 


0.0 


2 


85.8 


8.6 


16 


35 


79.9 


11.0 


9 


88 


0 


0.0 


1 


80.7 


10.7 


10 



Total: Mean - 84.2. SD » 8.9, N = 33 



49152 01 83.3 8.1 14 82.8 8.0 2 6 83.0 8.0 4 0 

02 78.5 7.4 1? 78.9 5.2 10 78.7 6.4 22 



Total: Mean * 81.5, SD * 7.7. N * 62 



(Continued) 
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Table 1 (continued) 




Descriptive Statistics for End of Course Examinations among January Enrollments 




during 1985 through 1968: N - 1649 


















Enrollment Year 










Test 




1985-87 


1988 




Total 


1985-88 


CDC 


Form 


Mean 


SO 


N Mean SO 


N 


Mean 


SO 


N 


61251 


01 


78.3 


7.7 


6 - 




78.3 


7.7 


6 




02 


84.7 


4.5 


3 




84.7 




3 




03 


78.0 


9.0 


12 84.5 9.2 


2 


78.9 


Q ft 

9.0 


14 




04 


85.6 


7.6 


18 84.0 0.0 


1 


85.4 


7 4 

/.4 






! 0 vB 1 1 


nean 


* ot.Z, 


SO « 8.3, N » 42 










Of Cf3 


01 


91.4 


5.3 


8 85.9 11.1 


9 


88.4 


0 ft 

9.0 


If 




02 


81.0 


19.8 


2 86.4 16.1 


7 


85.2 


15.8 


9 




Total: 


Hean 


- 87.3. 


SO * 11.6, H * 26 










74131 


27 


79. B 


9.1 


11 




79.8 


9.1 


11 




28 


77.2 


9.6 


12 




77.2 


9.6 


12 




29 


84.3 


9.0 


21 80.0 7.0 


7 


83.2 


8.6 


28 




30 


60.6 


9.3 


14 86.4 6.8 


9 


82.9 


8.7 


23 




Total: 


Mean 


« 81.6, SD « 9.0, 1* - 74 










Average of 


81.1 


3.8 


45 82.8 4.4 


28 


81.5 


3.7 


51 


Test Form Means 














Weighted Grand Mean 


« 81.31, 


SO - 9.029, N « 1649 











Table 2: Susmary of Analysis of Covar lance Examining the Effect of Scoring Treatment 


on End of Course Test Performance among January Enrollments: N 


- 1649 


Source R2 


SS 


df 


MS 


F 


P* 


Covariate .111 


14964 


51 


293 


3.907 


<.01 


Treatments . 000 


53 


1 


53 


<1.0 


>.25 


Error .888 


119338 


1596 


75 






Total 1.0* 


134355 


1648 








* Difference due to rounding from 5 decimal places 



The value of this finding Is somewhat ambiguous. If the point of interest is improving student performance, 
reducing turn-around time does not seem to be of much value. This finding is consistent with Hdmberg and 
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Schuemer and some of the research cited by Baker et a). On the other hand, If monetary costs are the 
principal Interest, the change did not adversely affect student performance and no change is, In fact, a 
desirable outcome. The Import of the findings are, then, dependent on the point of view of the reader. There 
may also be public relations aspects of quick turn around which would be of interest to some distance 
education providers. 



November Cancellations 

The results and conclusions of the January enrollments are replicated and confirmed by the November 
cancellations. The groups are considered Independent, although there was a slight overlap (under 5%) 
between the two. 

Comparing Table 3 with Table 2, the mean averages are all within one percent. The standard deviations are 
also close, and the relatively wider variability of the CY88 group over the CY85-B7 was maintained. The 
Analysis of Covariance of the November cancellation data (Table 4) Is also nearly identical with »nat shown in 
Table 2. In fact the difference In the R squared of the error term Is only .008. Test form continued to account 
for over 1 0 percent of the variability of test scores, while the effect due to VRE field scoring was not significant 
(p>.25). 
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Descriptive Statistics for End of Course Examinations among November Completions 
during 1985 through 1988: N * 1348 



Test 



1985-8? 



Enrollment Year 
1988 



Total 1985-88 



CDC Form 



Mean 



SD H 



Mean 



SD 



Mean 



SD 



11350 21 
22 
24 



83.3 
81.0 
76.0 



5.5 3 
14.1 2 
4.2 2 



83.3 5.5 3 
81.0 14.1 2 
76.0 4.2 2 



Total: Mean - 80.6, SD • 7.6. H * 7 



27132 


33 


78.4 


13.6 


20 








78.4 


13.6 


20 




34 


80.0 


6.7 


20 








80.0 


6.7 


20 




35 


73.3 


11.7 


3 


82.7 


7.9 


11 


80.7 


9.2 


14 




36 


70.0 


20.2 


4 


79.7 


8.3 


12 


77.2 


12.3 


16 




Total: 


Mean 


■ 79-1. 


SD » 


10.7. N - 


70 










30451 


33 


83. 1 


7.7 


12 


88.3 


5.8 


6 


84.8 


7,4 


16 




34 


86.1 


4.0 


13 


79.8 


9.8 


6 


84.1 


6.8 


19 



Total: Mean m 84.5. SD - 7.0. H = 37 



30750 25 


79.9 


10.7 


28 






79.9 


10.7 


28 


26 


85.3 


11.4 


27 






85.3 


11.4 


27 


29 








90.1 6.6 


7 


90.1 


6.6 


7 


30 








92.3 1.2 


3 


92.3 


1.2 


3 


Total: 


Mean - 


83.8. 


SD « 


11.0. K - 65 










32658C 23 


85.3 


8.6 


11 


85.3 10.5 


14 


85.3 


9,5 


25 


24 


84.7 


7.5 


21 


80.1 9.9 


17 


82.6 


8.8 


38 


Total: 


Mean » 


83 7. 


SD * 


9.1. N » 63 










32853 23 


78.9 


8.6 


21 






76.9 


8.6 


21 


24 


79.0 


8.1 


11 






79.0 


8.1 


11 


25 


79.7 


10.1 


49 


S..5 10.2 


15 


80.2 


10.0 


64 


26 


83.0 


8.8 


33 


83.6 8.5 


13 


83.2 


8.6 


46 


Total: 


Mean » 


80.9, 


SD « 


9.3. M « 142 
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Table 3 (Continued) 



Descriptive Statistics for End of Course Examinations among November Completions 
during 19B5 through 1988: N • 1348 



Enrollment Year 

Test 1985-87 1988 Total 1985-88 



CDC Form Mean SD H Mean SD N Mean SD ft 



Z3 


87.6 


8.0 


54 










87.6 


8.0 


54 


24 


85.3 


8.7 


61 










85.3 


8.7 


61 


25 








77. 


6 


4.2 


24 


77.6 


4.2 


24 


28 








79. 


7 


6.9 


21 


79.7 


6.9 


21 


Total: 


Mean « 


84.2, 


SD - 


8.5, n 




160 











42652 25 


79.3 


7.5 


63 






79.3 


7.5 


63 


26 


79.5 


7.7 


44 






79.5 


7.4 


44 


27 


79.8 


9.0 


76 






79.8 


9.0 


76 


28 


79.4 


7.4 


78 






79.4 


7.4 


78 


29 


80.8 


5.6 


9 


78.5 9.9 


42 


78.9 


9.2 


51 


30 


83.6 


5.1 


11 


80.7 6.4 


41 


81.3 


6.2 


52 


Total: 


Mean ■ 


79.7, 


SO = 


7.9. N - 364 











42753 01 








73.3 


15.0 


3 


73.3 


15.0 


3 


02 








60.7 


5.7 


3 


80-7 


6.7 


3 


21 


78.3 


7.4 


26 


71.5 


10.6 


2 


77.8 


7.6 


28 


22 


79.5 


7.8 


13 


75.7 


1.5 


3 


78.8 


7.2 


16 


Total: 


Mean » 


78.0, 


SD «= 


7.8, N - 


50 











43152 21 81.4 7.8 74 85.1 6.7 15 82.0 7.7 89 
22 80.1 9.3 62 85.4 6.2 16 81-2 9.0 78 

Total: Mean « 81.6, SD » 8.3, N « 167 



46450 34 


78-5 


13.0 


4 89.0 


0.0 


2 


82.0 


11.4 


6 


35 


79.5 


11.8 


4 78.0 


8.5 


2 


79.0 


9.9 


6 


37 






72.0 


0.0 


1 


72.0 


0.0 


1 


Total: 


Mean • 


• 79.6, 


SO - 10.2. N - 


13 











49152 01 82.8 8.4 24 81.6 7.4 14 82.3 8.0 38 
02 78.6 6.8 23 74.8 9.5 5 77.9 7.3 28 

Total: Mean - 80.4, SD - 7.9, N » 66 

(Continued) 
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Table 3 (Continued) 



Descriptive Statistics for End of Course examinations among November Completions 
during 1985 through 1988: N » 1348 







Enrollment Year 








Test 




1985-87 1988 




Total 


1985-88 


CDC Form 


Mean 


SO N Mean SO 


N 


Mean 


SD 


N 


61251 02 


78.0 


0.0 1 - 




78.0 


0.0 


1 


03 


78.1 


7.5 27 76.0 11.3 


7 


77.6 


8.2 


34 


04 


76.9 


9.9 24 82.0 0.0 


1 


77.1 


9.8 


25 


Total 


Mean * 


77.4, SD » 8.8. N » 60 










67273 01 


83.1 


6.1 9 91.2 5.3 


8 


66.9 


7.0 


17 


02 


95. 1 


6.5 8 90.0 12.8 


6 


92.9 


9.6 


14 


Total: Mean - 


89.6, SO « 8.7. H « 31 










74131 27 


76.8 


9.4 11 




76.8 


9.4 


11 


28 


76.6 


10.1 7 




76.6 


10.1 


7 


29 


80.6 


11.5 14 78.3 5.0 


3 


80.2 


10.6 


17 


30 


82.7 


9.0 17 100.0 0.0 


1 


83.7 


9.6 


18 


Total 


Mean « 


80.2, SD « 10.1, H « 53 










Average of 


80.7 


4,1 42 82.1 6.5 


31 


81.0 


4.2 


49 


Test Form Means 












Weighted Brand Mean « 


81.202, SO « 8.977, N « 1348 












Table 4: Summary of Analysis of Covariance Examining the Effect of Scoring Treatment 


on End of Course Test Performance among Hovember Completions: H 


* 1348 


Source 


R2 


SS df 


MS 


F 




f* 


Covariate 


.118 


12859 48 


268 


3.622 * 


.01 


Treatments 


.001 


57 1 


57 


<1.0 




>.2S 


Error 


,881 


95629 1298 


74 








Total 1.0 


108545 1347 
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Conclusion 

The elimination of Institutional scoring of formative volume review exercises and the implementation of VRE 
field scoring has made no significant difference In student performance as measured by summatlve end of 
course examinations. 
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Assignments for submission and turn-around time in distance education. 

A comment on Diehl 

Torstein Rekkedai 
NK1, Oslo 

Introduction 

When 1 started my research in distance education in the beginning of the seventies, ft was ^nly natural 
that two-way communication by written letters became one of the focn! points. 

Real two-way communication between the student and the tutor at a distance is considered to be one of 
the necessary requirements fen having a teaching activity within the definition of distance education (see for 
example Rumble 1989, Reagan 1988). When looking back on the experiment earned out in 1972 73, 1 find that 
I argued for the Importance of examining turn-around time: 

"For a Jong time yet we will have to depend on the mail, though in the future we can expect to be 
-hie to use other types of communication in distance education" (Rekkedai 1973, 1983). 

I was, of course, thinking of different uses of telephone communication, probably not at all foreseeing the 
rapid developments we experience today in applying computer mediated communication In distance 
education, developments in which I find myself dayiy occupied. 

Even today, I would be highly surprised if two-way communication by mail and questions about turn- 
around time as an important quality aspect of distance education should loose importance during the next few 
years. 

* 

Some early writings on reasons for dropping out of distance education courses pointed at problems 
concerning number and frequency of assignments and turn-around time. In a survey carried out by Sloan 
(1985) one of the reasons for dropping out Is that the time element Involved In communication with teachers 
lead to a drop in interest. Students had to wait long before their questions were answered by the teachers. In 
Sioan'ssurvey we also find that the students mention "reduce the number and extent of lessons" as a 
suggestion for Increasing -sompletfon rates. These viewpoints indicate the relationships between variables 
such as submission density, number of questions (or workload) per assignments, seif-check exercises and 
turn-around time. In another study by Harter (1969) the slow return of corrected assignments and 
administrative procedures were given as the most frequent reasons for dropping out Students In this study 
stated that they had waited from two to four months (!) for their corrected homework assignments. 
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The functions of the assignments for submission 

In my view, turnaround time Is of Importance to the students only as tar as the assignments for 
submission have functions which the students perceive as Important and to the degree that time Influences 
the quality of how these functions are taken care of. According to Grahm (1969) the assignments have two 
main functions; 

1. Learning efficiency can be Increased by demanding activity by the student and giving him/her guidance 
and structure In the study of course materials. 

2 Assignments enable the tutor to evaluate and follow the student's progress while the student Is informed 
about his/her progress. 

In a study by Baath (1976), several European correspondence schools (and the British Open University) 
ranked different functions of assignments for submission. The study clearly showed that according to the 
distance teaching educators the assignments for submission have functions of special relevance to tutors / 
Institute / administration on the one hand and to the students on the other hand, if we look at functions the 
assignments for the students have, the following aspects are of special importance (according to ranking In 
the study): 

a. To give the student effective feedback, help to correct mistakes and control their progress. 

b. To motivate the student - by serving as sub-goals, 
c To activate the student 

d. To give the student opportunities for application and transfer of knowledge. 

e. To counteract the students feelings of Isolation. 

9- To give the student help in survey, sum up and Integrate various parts of a course unit. 

h- To focus the student's attention on Important learning objectives. 

I. To serve as a means for the student to revise the unit. 

k. To teach In such a way that knowledge Is retained, through practice in writing. 

I. Sent according to schedule, to compel the students to regular work. 

One can argue that probably some functions in this list are more dependent on turn-around time than 
others. If so, turn around-time may be shown to be of different importance In relation to which functions are 
actuaBy perceived by the students to be of importance In their specific distance learning situation. 
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Research, experiments and generalisations 

Why do different researchers arrive at different conclusions? Can results from well designed and 
COrtroSed research be generalised? What can we team from practice? What can we team from research? 

During the last few years distance educators have searched for theories of distance education and for 
experimental research based on theories and deigned so that results may be generalised (see for example 
Moore 1985). I agree that we need theory buJding and experimental research to test theories. At the same 
time I am of the opinion that weH founded research based on Inductive methods aimed at finding solutions to 
practical problem? also contributes to developing knowledge, theory and better practice. Few facts In 
education can be universally generalised. This means that one might expect that research findings from well 
controlled and designed experiments should be put to the test In similar experiments with similar students 
under the same treatment Normally, however, teaching/learning situations differ - In many ways. Thus, one 
should not be surprised that d8ferent researchers reach different results. Nevertheless, I firmly believe that 
such research brings us forward In theory and practice, also when the results seem to be Inconsistent. 

Research on turn-around time 

In our preliminary research project at NKJ from 1973 we came to the following conclusions: 

1. There is a significant relationship between turn-around time and rates of completions. 

Z There is a significant relationship between turn-around time and number of assignments completed during 
the first three months of study. 

3. There is a significant relationship between turn-around time and the students' satisfaction with the time It 
takes to get the assignments back from the tutor. 

4. We found no significant relationship between turn-around time and the time It took to complete the 
course. 

5. We found no significant relationship between turn-around time and student performance measured by the 
final grades In the course. 

Since men relatfonsshlps between turn-around time and student performance and student attitudes have 
boon examined in many settings. In the ICDE sponsored international research project Barker et al. (1986) 
examined the relationship between some measures on studert-lnstltutJon contacts and persistence. They 
found no clear or consistent relationsships between turn-around time and persistence. The research showed 
statistically significant retetlonsship between these variables at only one of the five Institutions surveyed. This 
International research study also showed that there are great differences between the institutions and courses 
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concerning course length, average turn-around time, number of assignments, feedback intervals On this study 
time between feedback opportunities) and completion rates. The average turn-around time at the institutions 
varied from 8 to 25 days. These figures may be dtetnssed in relation to our findings and the findings of others 
(Baath & Mansson 1977), where the conclusions seem to be that the students seem to tolerate a turn-around 
time of about a week or less without negative effects, but not much more. Again, there is reason to believe that 
the limit would vary with courses, level of study, student groups, and even between countries. 

The British Open University has also collected some statistics and information on student opinions in 
connection with a try out of a new routing procedure for Tutor Marked Assignments" (Field 1987). 

The try out followed experimental procedures with a control group and an experimental group of tutors 
and students following the old and new route procedures. According to the old route procedures the tutors 
sent the graded and commented assignments through the main office for registration and checking, while the 
experimental students included a self adressed stamped envelope with the assignments to their tutors and 
received their assignments directly back from the tutor. As there is a fixed time limit for students to submit their 
assignments, and as the tutors are not allowed to return any assignments before the time limit Is passed the 
system Itself holds early submitted assignments back for some days. 

The experimental students received their assignments back 9 days earlier that the control group on an 
average (the averages being 24 and 15 days). The range of waiting times was 1.70 days in the experimental 
group and 1.87 days in the control group. The students In the experimental group were more satisfied with the 
wafting time. 

51 percent of the students who had experience of supplying the tutor with pre-stamped envelope said 
that it had been worth the Inconvenience, while only 36 percent of the control group answered the question 
positively. On a question whether It was (would be) worth the extra cost, the figures were 66 percentversus 40 
percent. 

In both the experimental group and the control group a majority wanted to receive their assignments 
directly back from their tutors. 67 percent of the experimental group preferred to get their assignments directly 
back, whfle only 9 percent preferred them to be routed through the main office. 

The USAF Extension Course Institute Studies 

I have read the three papers by Grover Diehl with great interest. I agree that the paper from 1982 supplies 
Interesting information about research unknown to many distance educators, myself included. 
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As Dlehl says, it has been taken for granted that Immediate feedback is instrumental for student learning 
and that feedback and reinforcement are more or less the same, as Information about correct answers and the 
satisfaction which results, function as reinforcement. 

it may be doubtful if feedback from tutors in distance-ducatton courses function as refnforcment as 
proposed in the behavourtst and Sklnnerian traditions. There has also during the last 10 to 15 years been a 
shift away from behavourist theory, step by step learning and study techniques based on learning as aquisltion 
of content rather that Increased understanding (see for example Marton 1979, Welngartz 1980). So. I welcome 
DiehJs viewpoints and his encouragement to distance educators to review and discuss the theories and 
experimental results concerning the "delayed retention effect". 

To many It might be a surprise that one of the elements often seen as a major drawback in distance 
education actually may have some "worthwhile educational advantage". 

On the oth jr hand, as Dlehl himself points out, most of the research on the delayed retention effect has 
been carried out In the traditional face-to-face setting, where actually "delayed" feedback is quicker than what 
we normally can achieve In distance education settings which base the two-way communication on postal 
services. Consequently, how a possible effeci of delayed feedback would work In our situation, and what 
would be the optimal delays for different learning tasks, would be interesting questions to go further into. 

DiehJ'8 conclusions for distance education from the literature on the delayed retention effect are clearly 
Interesting, and should result In discussion and possibly re-thinking of some assumptions that we may have 
taken for granted. Especially, his last point would mean a definite change In advice often given to distance 
tutors, that they should give feedback to all responses, also the very good ones, which corresponds to good 
theory in behavioural conditioning. 

However, when discussing turn-around time In distance education, feedback interval is only one element, 
and not necessarily the most significant one. Looking back on Bfiath's list above, only point 1 is directly 
concerned with feedback. 

Still, I can weU understand that Dlehl looks upon questions about feedback interval as an Important 
dimension in understanding possible effects of reducing turn-around time. 

If I understand Diehl's papers correctly, the ECI course volumes include in-volume tests Including quite 
small-step easy-earning tasks where Immediate feedback should be reccommended, while the erS of volume 
exercises are more complex and demand higher level understanding. So, the procedures followed of giving 
Immediate feedback on in-volume exercises and formal delayed feedback on the end of volume exercises 
should be well in line with the theory. 
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It is not completely clear to me whether the ECt courses are to be considered setf-fratructional material 
wfth formal Institutional control and feedback through multiple choice tests or actually distance education 
courses where the tests, scoring and feedback are supposed to function as assignments tor submission with 
clear teaching/learning functions. 

In my view, we can expect differences In how students experience turn-around time In relation to whether 
the assignments are supposed to function as didactic letters In communicating with a tutor (see for example 
HoJmberg 1983), or as end-oKnstruction tests where the feedback Is of a more formal nature. 

Or, using a different terminology, If we look at the distance student working In an Inner circle 
communicating with the material and an outer circle communicating with the tutor, turnaround time might be 
seen as a much more Important variable In systems were great emphasis Is put on processes taking place in 
the outer circle. 

As John Baath points out In his discussion of HoJmberg and Schuemer's paper I feel that the importance 
of quick tunvarouxf time is dependent on tutor behaviour and student-tutor relationships. In courses with a 
relatively high submission density and emphasis on tutor support in motivation, teaching and socialisation 
quick response from the tutor Is probably much more important than In systems with more impersonal or 
automatic feedback mechanisms. 

The two pepers measuring the effect of field scoring on completion time and student performance show 
that the new procedures had little effect 

in fact, these results ere not in disagreement with our findings on effects of reducing turn-around time. 
We ' tnti significant differences In completion rates, number of assignments submitted and student attitudes, 
but not in stud" *me and grades. 

Why UkJ rA the new procedures give statistically measurable results? From DteN's short pr.pers It is not 
quite clear which functions the end of volume review exercises have. From what I understand, the Career 
Development Courses at UASF EC1 differ In many respects from the distance education /correspondence 
study courses offered at the Institutions I know best, ;» seems to me that the volumes are quite large. They 
include In volume self-checking exercises, which might bee seen as substitutes for assignments for 
submission. The volume review exercises might be considered by the students as examination training. It 
might also be that the formal scoring is of little importance to the students, dependent on whether the results 
count toward the final grade and whether the formal feedback Is necessary for the student for assessing 
his/her performance. 

The Dlehl studies seem to be well designed and controlled. However, the research is not designed as 
controlled experiments. 
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The courses Include large variations In number of volumes and probably also subject, content and study 
objectives. Dtehl himself points end that the difference In treatment between the groups In the study not only 
Included the change from Institute to field scoring and reduction of feedback time, but "Simultaneously, 
enormous changes were being made In the entire Career Development course (CDC) program Including new 
Instructional format and automated course development". This fact may mean that other changes might 
counteract the effect of field scoring even if this change, if isolated, could have given measurable effect 

I also think that it should be noted that the result variables in the study are restricted to the measurable 
effects in completion time and student performance. Similar studies often collect student viewpoints and 
attitudes to the different treatments. In my view student attitudes are clearly relevant In assessing the total 
quality of the teaching learning system. 

Customer service and/or learning effects 

Dteni's conclusions are dear: 

1. ...the Im^' rotation of field scoring has opened a window for accelerated course 
completion, ^ .Mch students have overwhelmingly failed to take advantage. 1 ' 

2. The elimination of Institutional scoring of formative volume review exercises and the 
implementation of VRE field scoring has made no significant difference in student performance 
measured by summative end of course examinations," 

However, as Dtehi points out, as no effect is demonstrated the choice of procedures may depend on 
other considerations of practical, financial or pedagogical nature. Dieh! says: 

The importance of the findings are, then, dependent on the view of the reader. There may also 
be public relation aspects of quick turn around which would be of Interest to some distance 
education providers," 

To me this last point Is of prime Importance, and I would like to change the expression somewhat : the 
public relation aspects must be of focal interest to all distance education providers. (SarachelSBS). 

As distance educators we are working in the service Industry, whether w© represent private or public 
Institutions, whether we supply training for the open market, for companies and organisations or for our own 
employes. Concepts, theories and Ideas taken from service Industry market research and experiences are 
Important to us as are theories and experiences taken from the educational environment, in future we shall be 
competing with other offers to the student of recreation, education and other services. 

40 
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The demands on total quality Is Increasing. Student satisfaction is a result of acceptable relations 
between expectations and experiences. It seems dear to me that the demands on educational Institutions for 
higher quality and better service are Increasing. In the light of these developments turn-around time Is, 
according to students' viewpoints, one important element in the total qualliy concept. 

There Is also reason to believe that as electronic communication becomes more usual, students' 
expectations wffl change, and they wffl demand quicker feedback. While the students in the seventies were 
satisfHxfwlOTone-vreek return this wSI not be acceptable In 

the 90*168. Or while correspondence-course students accept one week, we have already experienced that 
students taking part In distance-education computer-based communication systems do not accept to watt for 
two days to have their questions answered, 

For some time distance-education systems that base their communication on different media and 
methods wffl exist side by side, and they wfll be compared. In this situation It Is my prediction that attention to 
turn-around time wBl be even more important than before. 
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Tutoring f equency in distance education - an empirical study of 
the impact of various frequencies of assignment submission 2 



Bdrfe Holmberg & Rudolf Schuemer 
FemUnlverslt&t/ZIFF, Hagen, FR Germany 



Summary / Abstract 



This is a study of the influence of assignment frequency or density (l.e., the distribution of 
assignments over the course units) and assignment length on the submission frequency, especially 
the completion rate, and the performance of learners. 

An English course was offered to FemUniversitflt students on a voluntary baste. The participants 
were divided into groups and received versions of assignments differing from one another in length 
as well as density. 

The hypothesis was that the pc. ~clpamj who were required to submit assignments to each course 
unit would show higher start and completion rates and would obtain better scores In the final tests 
than participants who were asked to send in the same assignments for submission but clustered 
after every two or four course units, Three degrees of assignment frequency were thus compared. 

The hypothesis was npj corroborated by the results: There are no significant differences hi the 
completion or success rates or in performance in the final tests. Furthermore, the results of an 
evaluation questionnaire demonstrate only slight differences between the groups. Possible reasons 
for this (non-) results are discussed. 



2 

An almost identical version of this paper was published as 'ACSDE Research Monograph No 1' at 'The American Center 
I for the Study of Distance Education, the Pennsylvania State University' in 19B9. 
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TUTORING FREQUENCY IN DISTANCE EDUCATION - AN EMPIRICAL 
STUDY OF THE IMPACT OF VARIOUS FREQUENCIES OF ASSIGNMENT 

SUBMISSION 



INTRODUCTION 

Questions asked, problems set and tasks given for students to answer, solve and cany 
out are the most common ways to initiate communication between students and tutors in 
distance education. They are usually referred to as assignments and require students to submit 
their solutions to the distance-teaching organisation, university or school. 

Whfle ft has been shown that quick handling of students' assignments exerts favourable 
influence on study results in that completion rates correlate with turn-around time (Rekkedal 
1983)', the equally reasonable assumption that frequent (non-contiguous) tutor-student contacts 
have similar consequences has not, or in any case only partially, been corroborated. 



Some reservations as to the general validity of these findings have been expressed after comparative 
studies of data from two distance-teaching organisations In Australia and one in each of the following 
areas: Canada, Pakistan, and the South Pacific: 

The results relevant to the examination of the relationship between turn-around time and persistence ... 
demonstrate no consistent trend even though the DDIAE Darling Downs institute of Advanced 
Education] data are consistent with Rekkedal's (1973) conclusion that low turn-around time is likely to 
increase persistence ... 

Of the 82 students who failed to complete requirements, 55 of these experienced high turn-around time, 
whereas of the 110 students who succeeded in completion requirements only 29 experienced a high 
turn-around, while 81 students had tow turn-around time, This pattern of results could be reasonably 
interpreted as pointing to the potential efficacy of turn-around time in influencing persistence. In the 
other four institutional contexts, however, there is no such indication of a significant statistical 
relationship, although the data for TSIT [ Gasman ian State Institute of Technology] were tending to be 
compatible with those of DDIAE, with 44 of the 70 students who failed to complete requirements 
experiencing high turn-around time, which could well have had a deleterious effect on student 
persistence. In the three other Institutional contexts, however, no such patterns emerged; rather, the 
results could be reasonably interpreted as being indicative of no salient relationship between turn- 
around and persistence" (Barker, Taylor, White, Giiiard, Khan, Kaufman and Mezger 1986, 30) 
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Previous research 

Dtatana«ducatlon practitioners and scholars have asked themselves what frequency 
exerts the most favourable influence on students' success. Thfe area was first approached In a 
scholarly way by Ufla Rosbefg-Johnsson as early as 1966. She designed a pJan for an empirical 
investigation which, however, was never undertaken (Rosberg 1966). A systematic study of the 
same problem has been undertaken by Baath who has made some Interesting experiments. He 
has examined three different groups studying the same course material with a view to finding 
Indications about what could be considered an Weal frequency of assignments ("submission 
density"). On the basis of the same instructional text one group was required to submit two 
assignments, a second group four and a third group eight assignments (in one experiment the 
number of submissions required of the three groups is three, six and 12 respectively, however). 
The total number of assignment questions was constant and the questions were Identical. Baath 
Biustrates the basic design of these experiments as follows: 

Assignment questions 
Submissions per submissi on total 

Group 1 2 x 48 - 96 

Group 2 4 x 24 = 96 

Group 3 8 x 12 » 96 

<?aath 1979. p.15; cf. Baath 1980) 



Baath's study comprises achievement variables, study-time variables and attitude variables. 

In an ©arty report on his findings he pointed to one Very clear experimental result": more 
students In the groups with a greater number of submissions "began sending in papers than did 
the students in the other experimental groups". 

No corresponding consistent differences between the experimental groups were found 
with regard to course completion. The proportion of completers was about the same in 
the lew, medium, end high submission density groups in most of the experiments, Nor 
wfth regard to final test results was It possible to find any significant differences. 

(Bairn 1979, 17-18; cf. Baath 1980). 
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Significant differences as to attitudes could be established, however. Higher submission 
density was shown to correlate with "more positive attitudes to the assignments for submission* 
and to the overall appreciation of the course of study. Students' attitudes to the submission 
density varied in a way to be expected hi mat "high-density groups were negative to an increase 
of the number of submissions, whereas low-density groups favoured such an increase, and 
medium-density groups were uncertain" (B&ath 1980. 151). 

it Is disappointing that Baaih's study, which was a painstaking one performed with great acumen, 
should have resulted in no more clear-cut results than those mentioned. It seems natural and 
expected that more students should complete and submit assignments to s short first unit than to 
a long one, which is all that the BA&th study on achievements definitely confirms. 

Most educators probably agree that goals dose at hand: I.e., goals that can be attained 
In a reasonably short time, are motivating in that they demonstrate to the student that he or she 
makes progress. If motivation is taken to promote success this would seem to indicate that a 
suitably high submission frequency must be expected to lead to greater success than low 
submission frequency, provided, of course, that the assignments and the units leading to them 
are felt to represent steps on the path to the desired competence. This was the main hypothesis 
of the present study. 

We further assumed that the difficulty and size of the assignments in the sense of work 
and time required to carry them out would influence the learning and students' readiness to finish 
a course. Too comprehensive as well as too short and bitty assignments may discourage 
students from completing a course. 



The basis of the FernUnlversltat study 

Against this background It was felt to be of interest to bring about a kind of replication of 
Berth's study. This was done In the years 1987/88 at the FemUniversltfit Institute for Research 
Into Distance Education (ZIFF). 

As in the West German university situation there are legal problems connected with 
offering students of regular degree programmes alternative versions of a compulsory course; 
therefore, a special course was developed for the study and offered free of charge to 
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FernUnlvefBltfit students Interested in taking It as a voluntary additional part of their study. The 
course (Hdmberg 1986) is a fairly elementary one on English language proficiency at a level 
assumed to be relevant to university students of other subjects than English. Its title Is "Essentials 
of English". It consists of 14 course units (328 pages of the size of the pages of this report), an 
audtocassette containing 60 minutes of English speech, exercises for self-checking and extensive 
assignment tasks based on each of the 14 units, 



51 



METHOD 
Design 

Two (actors (F1 , F2) were to be varied In the study: 

F1 : The number of Hems; t.e., the length of the assignments . 

Three levels: short assignments (L« 1), assignments of medium length (L=2) and long 

assignments (L«3). The ratios between the levels should be 1 :2 : 4 
F2: Distribution of the Items over the course units or 'density* In Baath's terminology. 

Three levels: assignments after each course unit (D=1), after every two course units (D= 2) or 

after every fourth course units (D = 3) , 

It seems not to be practicable to cross the factors completely (Le., to combine each level of 
facto- 1 with each level of factor 2) because then the result would be some assignments with too 
heavy a workload (e.g., for Jong assignments - L=3 - after every two or four course units - D « 2 or 
D=3). 

Therefore, only six experimental conditions or combinations of the two factors were 
realized, Le., the assignments versions A-F : 
* Version A: long assignments after each course unit 
(combination of L= 3 and D«1) 

- Version fit assignments of medium length after each course unit 
(combination of L=2 and D= 1) 

- Yerston Q: short assignments after each course unit 
(combination of L« 1 and D= 1) 

- Yergion P: assignments of mt Sum length after every two course units (combination of U2 
and D =2); the items of the assignments In version D are Identical In content and length with 
those of version B 

- Version E : short assignments after every two course units (combination of 1^-1 and 0=2); the 
Items of the assignments in version E are Identical In content and length with those of version C 

- Version F: short assignments a?er every four course units (combination of L« 1 and D =3); the 
ttems of the assignments in version F are identical in content and length with those of version C 



48 



Holmberg & Schuemer: Tutoring frequency .../Method/ 52 



A group of course participants were assigned at random to each of the experimental conditions 
or combination of length" and "density* (versions A-F). This results in the design shown in Box 1 : 



Box X: Design 


Length (D) 


Density: assignments after 
each every every 

CU two CU four CU 
D=l D=2 D=3 


short (L=l) 
medium (11=2) 
long (L»3J 


G3 G5 G6 

G2 G4 

Gl 


* CU: course unit(s) 



in addition, a control group was to be studied with a view to finding out whether and to 
what extent the participants make progress by studying the course. This group is not included in 
the design above. The assignments for this group correspond to those of group 3 (version C) 
with the difference that this group was required to complete the assignments for the last two 
course units twice: (a) before the beginning of the course (pretest), and (b) at the end of the 
course (posttest). 

The interaction between the factors "length" (I) and "density" (D)' cannot be tested with all 
groups because of the incomplete crossing of the factors, but it is possible to test this interaction 
by means of the data of the groups 2 - 5: G3 with D=1/L=1; G5 with D=2/I-1; G2 with 
D-1/L-2; and G4 with D=2/L»2. 

Pependent variables 

The following (dependent) variables were studied in relation to "density" and length* as 
independent variables: 
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- Starter rate: percentage of those submitting assignments a! the first ooportunlty ior submission 
fcouree unit CU 1 for groups 1,2 and 3; CU 2 for groups 4 and 5; CU 4 for group 6) - related to 
the number of enrolments In each group. 

- Prop-out rate: percentage of those who stopped submitting assignments before finishing the 
course. 

- Comptefenraty percentage of those finishing the course (completing the last two 
assignments) 

* Achievement scores for the last two assignments 

Some additional variables were to be recorded by means of three questionnaires: 

- Attitudes to the course and Its material (Evaluation questionnaire) 

- Reasons for not submittin g anv assignment (Non-Starter questionnaire) 

- Reasons for refraining from submitting assignments before the end of the course 
Prop-Out questionnaire) 



Hypotheses: 

(1 ) More frequent contacts between tutor and learner will result In 

- a higher submission rate {i.e., a higher start and completion rate and a lower drop-out rate) 

- better achievement scores 

- a more positive evaluation of the course by the participants. 

Thus assignments after each course unit (D = 1 ) were expected to have more positive effects 
in the sense described than assignments after every two or every four course units (D=2 or 
D*3). 

(2) Assignment tasks of greater length were (up to a certain upper limit) expected to have similar 
effects as more frequent tutor-learner contacts; Le., 

- a lower drop-out rate 

- a higher completion rate 

- better achievement scores 

- a more positive evaluation of the course by the participants. 

Assignments of greater length (L«=2 or L=3) were thus assumed to stimulate the learner to 
study the course material more Intensively and to give the learners more opportunities for 
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(mediated) learner-tutor communication and, therefore, lo be more effective than shorter 
assignments. 

But there are restrictions as to the relevance of this hypothesis: The optimal length of 
assignments obviously depends on the course (eg., Its subject matter, its learning 
objectives, its content and its complexity) and probably also on the individual learner (e.g., 
Ns/ner previous knowledge, learning style and his/her time budget for studying). It seems 
plausible that longer assignments In a language course as used In this study will give more 
opportunities to exercise and practice than shorter ones, particularly as the subject matter of 
the course Is rather easy to grasp and hardly causes great problems of understanding but 
rather aims at making students exercise and practice English usage. Too short assignments 
imply the risk of demanding too little. 

It seems not to be unreasonable to expect that the effects of the "densitiy" factor is not 
the same on ail levels erf the length" factor or that the combined effects of both factors on 
achievement, for instance, are non-additive (non-addftfvity or interaction of the factors D and L). 
But the form or shape of this possible Interaction may be dependent on the course (its subject 
matter and difficulty, etc.) similarly as with regard to the effect of the length" factor. No specific 
hypothesis regarding the form of this Interaction has been developed, but this interaction can be 
tested by means of the data of groups 2-5 (see above). 

Testing the hypotheses - statistical procedu res 1 
(a) With regard to frequency o f submission 

The hypotheses regarding the submission frequencies were tested by means of Chi 2 . 
The frequencies of submissions for the last two course units (or the completion rates) In the 
experimental groups 1-6 were compared and tested by a "conventional' Cnt 2 (and by a parallel 
test according to the logllnear model). 



i 

All analyses were carried out by means of the Statistical Analysis System (SAS - see references) 
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The starter rates (frequencies of participants submitting • -slgnments at the first 
opportunity) can be compared directly only between the groups 1-3 (with assignments of varying 
length after each course unit). 

In addition Chi 2 values were computed for the submission frequencies of the groups for 
each course unit but these values are Interpreted In a purely descriptive way. 

(b) Regarding the achievement scores 

The differences between the six experimental groups In the achievement scores for the 
assignments belonging to the last two course units were tested by means of a univariate analysis 
of variance with the factor "groups' as the Independent variable and the achievement scores 
(sum of raw points over items) as the dependent variable. Only If this overall test of the factor 
"group" Is significant In the expected w^ y is it meaningful to carry out additional 'planned 
comparisons" for the effects of the factors length" (L) ami "density" (D) and their Interaction 
(L*D), on the basis of the data of the groups 2-5. 

(c) With regard to course evaluation 

The Items of the evaluation questionnaire were reduced to composite scores {factor 
scores) by means of factor analysis. Each of these scores Is uted as the dependent variable In a 
univariate analysis of variance to compare the experimental groups. 

A significance level of alpha =.Q1 Is used for all statistical tests. 
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Carry.ng out the study 

(a) Description of the course: 

The course used for the study was an English language course "Essentials of English" 
(Hofmberg 1986). The course was not part of a degree programme and the participation was 
v^untary. 

The course comprises 14 course units. Each unit contains about IS pages of 
Instructional text and between five and 10 pages of vocabulary and commentary; in addition each 
ur.it contains one or two setf-chr *tdng exerciser with 2-5 Hems. The course material Includes an 
audiotape (speakers: the course author and two "native speakers"). The course was announced 
tai an Information booklet of the FernUniversltat as an English course for adults with some 
previous knowledge (minimum: about four years of school teaching). The level may be 
characterised as that of the German "gymnaslale Oberstufe". 

A certificate of participation was promised to learners completing the course (t,e., 
submitting assignments to all course units belonging to their programmes). 

The course began In the spring of 1 987 and was planned for a maximum running time of 
a year. Later three additional months for submitting assignments were conceded to the learners. 
There was no time schedule prescribed for submitting the assignments with the exception of the 
final date. 

The assignments submitted by the learners were corrected and commanted on by two 
teachers of English engaged especially for carrying out the project. The turn-around time for the 
correction of and commenting on the assignments varied to some degree with the workload of 
the two tutors; but with few exceptions it was possible to keep a turn-around time of 10 days or 
less. 
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(b) Description of the types of Items In the different versions 

Each assignment version corresponded to one experimental condition and was assigned 
to one of the experimental groups 1-6. A seventh group functioned as control group (see 
Group 7). 

The groups 1-5 were asked to submit assignments after each course unit, the length of 
which varied approximately in the ratio of 4 : 2 : 1 in the groups 1, 2 and 3. 

QfVUP 1 received version A. The assignments for the course units 1-7 of his version 
consisted of 8 items each (mostly short sentences to be translated from German into English or 
vice versa, and lists of words to be arranged by rhyme or for which the phonetic transcriptions 
were to be given). 

The assignments for the course units 8-12 consisted of four translation Items each (two from 
German Into English and two vice versa); each item or text to be translated comprised two 
paragraphs and had a length of about one fourth page. 

GfOUP 2 received version B . The assignments for the course units 1-7 of this version 
consisted of four Heme each (mostly short sentences to be translated from German into English 
or vice versa, and lists o* words to be arranged by rhyme or for which the phonetic transcriptions 
were to be given). 

The assignments for the course units 8-12 consisted of two translation Herns each (one from 
German Into English and the other vice versa); each item or text to be translated comprised two 
paragraphs and had a length of about one fourth page. 

Group 9 received version C . The assignments for the course units 1-7 of this version 
consisted of two hems each (mostly short sentences to be translated from German Into English 
or vice versa, and lists of words to be arranged by rhyme or for which the phonetic transcriptions 
were to be given). 

The assignments for the course units 8-12 consisted of two translation items each (two from 
German Into English and two vice versa); each Item or text to be translaled consisted of a 
paragraph and had a length of about one eighth page. 

The groups 4-6 had to submit assignments with the same items as the groups 2 and 3, 
repectively. but with different submission density. Group 4 (version D) had ihe same items as 
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group 2 (version B) in that the items of two assignments in B were put together to form one 
assignment in D Group 5 and 6 (versions E and F) received the same Kerns as group 3 (version 
C); the items of two (group 5, version E) or of four (group 6, version F) Items of version C are 
combined to form one assignment In E or F. 

QroujM received version D. This version corresponds to version B for group 2 with the 
difference that the items of two subsequent assignments of B were put together to one 
assignment In version D, So the first assignment of version D (to be submitted course unit 2) 
corresponds with the first two assignments of ve~'on B; the second assignment of D (to be 
submitted after course unit 4) corresponds to assignments 3 and 4 in version B, and so on. 

Group? received version E. This version corresponds to version C for group 3 with the 
difference that the Items of two subsequent assignments of C were put together to one 
assignment In version E. So the first assignment of version E (to be submitted after course unit 2) 
corresponds with the first two assignments in version C; the second assignment of E (to be 
submitted after course unit 4) corresponds to assignments 3 and 4 of version C, and so on. 

Group 6 received version F . This version corresponds to version C for group 3 with the 
difference that the items of four subsequent assignments of C were put together into one 
assignment in version F. So the first assignment of version F (to be submitted after course unit 4) 
corresponds with the first four assignments of version C; the second assignment of F (to be 
submitted after course unit 8) corresponds to assignments 5 to 8 of version C, and so on. 

Group 7. the control group, received the same assignments as group 3 {version CI with 
the difference mat the members of this group had to complete the assignments for the test two 
course units (1 3 and 14) twice: (a) before beginning the course (pretest), and (b) at the end of the 
course (posttest). 

The assignments after the last two course units (13 and 14) were Identical for all groups. 
They aimed at repetition with respect to the learning objectives of the course as a whole. 
Therefore, achievement scores in these assignments can be used as dependent variables In the 
tests of the achievement hypotheses. 
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The assignment after course unit 13 consists of two comprehensive translation Items, 
and three grammar exercises (referring to tenses and the wording of questions as well as a task 
Implying answering questions In English). 

The assignment after course unit 14 consists of two comprehensive translation Items, three 
grammar (plural forme of nouns, verb forms) and a further translation exercise. 
(Further details about the course and the assignments can be found in HoJmberg, Schuemer et 
a). 1988). 

(c) Number of enrolments 

Student enrolments totalled 1269. Students were assigned to the groups/experimental 
conditions at random; the number of enrolments per group were: 

Group (Gl) : 150 
Group (G2) : 155 
Group (G3) : 150 
Group (G4) : 151 
Group (G5) : 151 
Group (G6) : 149 
Group (G7) : 138 

The somewhat lower number of enrolments for group 7 (control group) Is because ft 
could not be foreseen at the beginning how many students would enrol; therefore, the 
distribution of the applicants to the groups was at first restricted to the experimental groups 1 -6. 

fd) Dispatch of the course material 

The dispatch of the course material and the assignments was made dependent on the 
individual student's submission behaviour (i.e., assignment submission was a condition for 
further course material being sent to students after the initial delivery.) 
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(e) Questionnaires, 

As mentioned In section 2.1 the participants were Invited to express their views in three 
questionnaires about: 

- the quality of the course and the tutoring (course evaluation) 

- the reasons why they did not submit any assignment (questionnaire for non-starters) 

- the reasons why they stopped submitting further assignments (drop-out questionnaire). 

The evaluation questionnaire includes 43 statements on the quality of the course 
material, Its structure and learning objectives, the assignments, the correction of and the 
commenting on the assignments, etc. 

The non-starter and the drop-out questionnaires consist of several Items referring to 
possible reasons for refraining from submitting assignments and drop-out, respectively. 

Most of the Items in all three questionnaires mentioned have the form of statements 
(Ukert type) where the Subjects (Ss) were asked to respond to each statement on a 4-polnt scale 
of agreement: with 0, not applicable for S; 1 , little; 2, largely; 3, completely applicable. 

The evaluation questionnaire was sent to the participants together wnh the material for 
the second-to-Jast course unit. The non-starter questionnaire was sent to those students who had 
applied for enrolment but had not submitted any assignment before the middle of January 1988; 
the drop-out questionnaire was sent to participants who had submitted assignments only up to 
course unit 7 two weeks before the final date. 



ffl letters of encouraoernflnt 

A letter of encouragement was sent to course participants who had not submitted any 
assignment as late as tb* end of October 1 987. 

A simflar procedure was applied to those who had at first submitted some assignments 
but then had not been heard from as late as the end of July 1987. 
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ft should be noted that such an encouraging procedure tends to reduce the effects of the 
different conditions/ treatments. But & seemed to be meaningful to study the effect of density and 
length under most realistic conditions: Many distance-teaching institutions use some form of 
encouragement to reduce non-starter and drop-out rates (in most cases standardised or 
Individual tsed tetters; cf. Schuemer, section 3.5.1 in Graff and Holmberg 1988). 



(g) Certtflcates 

The course under study is no part of a degree programme. The participants who had 
submitted assignments to all course units belonging to their programmes were issued a 
certificate of participation (without any further qualification). 
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RESULTS 

First the submission frequencies, then the achievement scores (for the assignments of 
the lost two course units) and, finally, the results of three questionnaires win be described. 



Submission frequency 



Table 1 contains the submission frequencies for each group and for each course unit 

(CU). 



Table 


1: Submission 


frequency for each 


group 


and each 




course 


unit 


(CU) 








N A : Number of participants enrolled 




Group: 














l 


2 


3 


4 


5 


6 


7 




N A 150 


155 


150 


151 


151 


149 


138 


CU 
















1 


96 


111 


110 








78 


2 


79 


94 


104 


90 


105 




74 


3 


67 


86 


74 








53 


4 


51 


73 


63 


66 


81 


55 


46 


5 


48 


58 


56 








39 


6 


42 


55 


50 


60 


67 




34 


7 


35 


51 


44 








31 


8 


30 


46 


40 


50 


56 


38 


32 


9 


24 


43 


37 








29 


10 


20 


39 


33 


42 


46 




28 


11 


18 


32 


32 








25 


12 


16 


31 


28 


34 


40 


26 


25 


13 


13 


27 


25 


31 


34 


22 


23 


14 


13 


26 


20 


28 


28 


21 


23 



The corresponding submission rates (frequencies related to the number of participants 
enrolled) for each group (1-6) are summarised in Table 2. (Group 7, which has the only function 
of testing the learning progress In a before-after comparison, will not be considered in this group 
comparison.) 



i 



59 



Hotmberg & Schueman Tutoring frequency ... /Rwufts/ 63 



Table 


2: 


Submission rates for 


groups 


1-6: 






?a • 
% s 


Number of participants enrolled 
Percentage related to M A 












Group: 
1 2 


3 


4 


5 


6 






N A 


150 


155 


150 


151 


151 


149 


rate 
(Gl-6) 
Ln percent 

1 (%) 


CU 1 


% 


64.0 


71.6 


73.3 










CU 2 


% 


52.7 


60.7 


69.3 


59.6 


69.5 


mm 




CU 3 


% 


44.7 


55.5 


49.3 










CU 4 


% 


34.0 


47.1 


42.0 


43.7 


53.6 


36.9 


42.9 


CU 5 


% 


32.0 


37.4 


37.3 










CU 6 


% 


28.0 


35.5 


33.3 


39.7 


44.4 


— ] 




CU 7 


% 


23.3 


32.9 


29.3 










CU 8 


% 


20.0 


29.7 


26.7 


33.1 


37.1 


25.5 | 


28.7 


CU 9 


% 


16.0 


27.7 


24.7 










CU10 


% 


13.3 


25.2 


22.0 


27.8 


30.5 






CU11 


% 


12.0 


20.7 


21.3 










CU12 


% 


10.7 


20.0 


18.7 


22.5 


26.5 


17.5 | 


19.3 


CU13 


% 


8.7 


17.4 


16.7 


20.5 


22.5 


14.8 | 


16.8 


CU14 


% 


8.7 


16.8 


13.3 


18.5 


18.5 


14.1 | 


15.0 
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The starter rate 1 for groups 1-3 (Q1-G3) is rather high (69.7%); also the starter rates for 
groups 4 and 5, which had to submit no assignments until they had completed the second 
course unit (CU), are comparatively high (04: 596% ; G5: 69.5%). 

As only groups 1-3 had to submit assignments (of different length) for course unit 1 , only 
their starter rates should be compared directly. Group 1 with the longest assignments (L=3) 
yields a lower rate than the two other groups with the medium and shortest assignments (62: 
L-2; 03: L»1). But the differences between the groups tested by the Cb^-test are not 
significant 2 : Chi 2 =3.5; rfi=*2; p £.17; contingency coefficient CC«.09. 

A comparison of the starter rates In groups G4 and G5, which had to submit their first 
assignment after finishing course unit 2, with the starter rates for the second assignment in 
groups 1-3 shows: 

- group 1 with the longest assignments (L=3) has the lowest rate (52.7%) 

- groups 2 and 4 with assignments of medium length (L=2), but different submission density 
(G2: assignments for each course unit; D«1 / G4: assignments for every two course units 
D= 2) have rates near 60% 

- groups 3 and 5 with the shortest assignments (L=1), but different submission density (G3: 
assignments for each course unit; D= 1 / G5: assignments for every two course units D= 2) 
have the highest rates (near 70%) 

These differences are significant: Chi 2 - 13.1 ; df =4; p <01 ; contingency coefficient CC« .13. 

Considering the rates for the assignments belonging to course unit 4 allows a 
comparison of all the six experimental groups: 

- group 5 with short assignments for every two course unit (l=1/D=2) has the highest rate 
(54%) 

- the rates for groups 2 and 4 with assignments of medium length for each course unit (G2: 
L«=2/D=1) or for every two course units (G4: L=2/D=2) are the next highest (47.1% and 
43.7%) 



1 

Ait rates are related to the number of participants enrolled. 

2 

*Conventk>naf Chr^-test; an anaJoguous test by the togjnear model yields similar results. This holds also 
lor the other Chr-tests referred to: in no case does the Jogltnaear test yield results that contradict the 
conclusions suggested by the conventional Chi -test. 
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- group 3 with short assignments tor each course unft (L=* 1/D-= 1) yields a rate of 42% 

- group 6 with snort assignments for every four course units (L.= 1/D=3) achieves a rate of 37% 
which is equally low as the rate of group 1 with assignments of the greatest length (L=3)for 
each course unit (L*3/D«=1): 34% 

These differences are significant: Chi 2 - 15.3; df=5; p <.01; contingency coefficient CC=. 13. 

Similar relations between the groups (lowest rate for group 1 and highest for group 5) 
can be found also for course units 8 and 12 as well as for the assignments of the two final course 
units 13 and 14 (although there is a tendency towards a lower degree of difference with the later 
course units). 

The submission frequency decreases continuously and drastically from the first to the 
last assignment in all groups: for example from above 70% for the first assignment in groups 2 
and 3 to below 20% for the last assignment or from above 60% to below 10% In group 1 . 

The submission rates for course units 13 and 14 with identical Items for all groups can be 
Interpreted as completion rates. These rates are very low 3 in all groups: only 16,8% for course 
unit 13 and 15% for course unit 14 (overall rate by summing up the submission frequencies over 
the six groups and relating this sum to the overall number of enrolments in the six groups). 

A comparison of the completion rates for the 6 groups In a purely descriptive way shows 
the following tendencies: 

• Group 1 (with the longest assignments for each course unit; L=3/D= 1) yields the lowest rates: 
8.7% for course unit 13 and 14. 

- Group 5 and 4 (with short or medium assignments for every second course unit; G5: 

L= 1/D=2; G4: L=2/D=2) have the highest rates (G5: 22.5% for course unit 13 and 185% for 
course unft 14 / G4: 20.5% for course unft 13 and 18.5% for course unft 14). 



For comparison: In an international survey of 197 institutions a median completion rate of about 67% for 
the three courses with the highest numbers of enrolment at each institution is reported (for further details 
see Schuemer, section 32.10 In Graff and Hdmberg 1988). The above-mentioned overall completion rate 
of 15% is even tower than the completion rates for other courses of the FemUnhrersitat, which are 
between 20% and 30% In most cases (cf. Doerfert. Schuemer et at. 1988, 196) 
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- Group 2 (with assignments of medium length for each course unit; U2/D=1) yields rates 
somewhat lower than those of groups 4 and 5: 17.4% tor course unit 13 and 16.8% for course 
unit 14. 

- The rates for group 3 (with short assignments for each course un&; L=1/D=1) and for group 6 
(with short assignments for every four course units; L=1/D=3) are lower than those for group 
2 and higher than those of group 1. 

Arranging the groups according to the length of assignments and their density shows: 

- If assignments are to be submitted for each course unit (G1 - G3) the completion rate for the 
group with assignments of medium length (G2) is higher than that of the group with short 
assignments (G3) and also higher than that of the group with the longest assignments (G1 ). 

- A comparison of the groups which had to submit assignments of Identical length and content 
but with varying density of submission (a: G2 vs. G*; ,/fth L=2 and D=1 or 0=2 and Jj: G3 vs. 
G5 orG6 with L=1 and D*1 or D=2 or D=3) shows: if the assignments have to be submitted 
after every two course units then the completion rate is higher than If the assignments are to be 
submitted after each course unit or after every four course units, This tendency Is stronger for 
short assignments (G5 vs. G3 or G6) than for assignments of medium length (G4 vs. G2). 

AD the differences or tendencies described above are rather weak, however. An overall- 
Chr 2 test for the submission frequencies In group 1 - 6 is noj significant (either for course untt 13 
or for course unit 14; see Table 3): 
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Table 3: Comparison of the submission rates for course 
units (CD) 13 und 14 in groups 1-6: 


tt_fl i 

% 

CC J 


: Number of enrolments 

: Frequency of submission 

: Frequency of non-submission 

; Percentage related to N»; submission rate 

s Contingency coefficient 


group; 

1 .3 4 5 6 




N A 150 155 150 151 151 149 | Chi 2 df CC 


CU13 »+» 13 27 25 31 34 22 
137 128 125 120 117 127 
% 8.7 17.4 16.7 20.5 22.5 14.8 


12.6 5 .12 
p£.03 


CU14 "+ w 13 26 20 28 28 21 
137 129 130 123 123 128 
% 8.7 16.8 13.3 18.5 18.5 14.1 


8.5 5 .10 
p<.13 



The conclusion is: The main hypothesis - that a higher submission density yields higher 
completion rates - cannot be confirmed by the results (on the H 0 of "no differences between the 
outcomes of the differing experimental conditions' cannot be rejected). 

As the overall test for the differences between the six groups is not significant ft Is not 
meaningful to carry out further special tests for comparisons w*>i regard to the factors length" 
(L) or "density" P). 



Achievement 

The assignments for course units 13 end 14 are identical in ail experimental groups and 
refer to the learning objectives of the course as a whole. Therefore, the achievements in these 
assignments can be interpreted as a final test with regard to these objectives. 

The Inter-rater consistency of the assessment of the achievement scores should be 
considered first before the comparison of the achievement scores between the groups. 
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Nearly all of the assignments for course units 13 and 14 submitted by the learners were 
assessed Independently by the two tutors. The scores for each assignment given by either of the 
tutors were compared and correlated (Spearman's rho). The correlation coefficients for the 
overall scores (sum of points over Items for the assignment of each course unit: AS13 and ASH) 
are rho«=.94 for AS13 and rho =.96 for AS14. Similarly high coefficients have been found for each 
of the single Hems in the assignments: coefficients between rho =.86 and rho -.98. Furthermore, 
the differences In the median or mean scores between the two raters are very small. Therefore, It 
can be concluded that the two tutors agree very consistently in their ratings. 

in addition It should be tested whether the learners have made some learning progress 
by studying the course. This can be done by means of the data from the 'control group" 7 which 
completed the assignments for the course units 13 and 14 twice: (a) before the beginning of the 
course (pretest), and (b) at the end of the course (posttest). 23 of the 103 learners of group 7 
who participated In the pretest havr ilso submitted assignments to the course units 13 and 14 at 
the end of the course. The differences between posttest and pretest scores can be compared by 
means of a t-test (difference test for correlated observations). These tests yield highly significant 
posttest-pretest differences: t=6.54 for the assignments of course unit 13 and t=5.36 for course 
unit 14 (N«23; P<001 In either case). Therefore, ft can be concluded that the course is efficient 
to some degree; the learners have made some progress by studying the course. 

Only the achievement scores of the six experimental groups 1-6 are considered In the 
following discussion 

Table 4 contains the mean achievements scores (M) and their standard deviation (s) for 
each of the groups 1-6, which are arranged according to the levels of the factors length" (l) and 
"density" (D). The achievement scores are simply the sum of the points given by the tutors for the 
Items In the assignments for course unit 13 (AS 13) and couise unit 14 (AS14). (AS13+ 14 means 
the sum of the scores for AS 13 and AS 14). 
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Table 4: Mean achievement scores (M) and standard deviations (s) per 
group for the assignments to course unit 13 (AS13) and course 
unit 14 (ASM) and for the sum (AS13+14) 



Density levels: D»l: assignments for each course unit 

0*2: assignments for every two course units 
D»3: assignments for every four course units 



Length (L) 




density: 


D=2 


D=3 


AS13 










short (I*»i) 




G3 


G5 


G6 




N 


56.4 


54.6 


56.2 




s 


8.7 


10.0 


9.8 






(n=25) 


(n=34) 


(n=22) 


medium (L=2) 




G2 


G4 






M 


53.4 


55.6 






s 


9.6 


10.5 








(n«27) 


(n=31) 




long (L=3) 




Gl 








N 


50.7 








8 


9.9 










(n-13) 







continued: s. next page 
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Table 4: continued 






1 

1 


AS 1.4 












short (li-l) 




G3 


G5 


G6 




M 


59.6 


60.5 


60.0 






s 


8.8 




ft (S 








(n^O) 


(n=28) 


(n«2l) 




medium (L=2) 




G2 


G4 








M 


58.5 


61.4 








s 


8.7 


8.2 










<n=26) 


(n«28) 






lona fl"j=s3^ 




Gl 










M 


58.5 










s 


9.2 












fn«=13) 








AS13+14 (Sum 


of 


AS13 and AS14) 










D=l 


D=2 


D=3 




short (L=l) 




G3 


G5 


G6 




M 


116.2 


115.5 


117.0 






s 


16.0 


17.9 


Id. J 








(n~20) 


(n-28) 


(n=21) 




medium (L=2) 




G2 


G4 






H 


M 


111.6 


117.3 






M 


s 


17.0 












(n=26) 


(n=28) 






long (U=3) 




Gl 








M 


M 


109.2 








M 


s 


16.6 








1 




(n=13) 









The differences between the mean scores are very small (in relation to the great variation 
of the scores within each of the groups; see Table 4). This Is true with regard to the scores frr the 
assignment of course unit 13 (ASi3), as well as for the assignment of course unit 14 (AS14), and 
for their sum (AS13+14). And It Is also true if the group comparisons are restricted to those 
groups which had to submit assignments of Identical content for course units 1-12. but In 
different density: So the mean scores of the groups 3, 5 and 6 (with short assignments (L=1) 
after each course unit (G3: D«1) o ifter every two course units (G5: D*2) or after every four 
course units (06: D«3) ) are close together; similarly, the differences between the groups 2 and 4 
(assignments of medium length (L=2) after each (G2: D= 1) or after eve»y two course units (G4: 
D=2) ) are small. The differences between all the six groups are too small to be of any practical 
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or statistical significance. Unlfactorial analyses of variance with the factor "groups* as the 
Independent variable (6 levels) aid the scores for AS 13 or AS14 or for both together (AS13+ 14) 
as the dependent variables show that the differences between the groups are ryj. significant 4 
(see Table 5). 



Table 5: Comparison of the achievement scores of the 6 experimental 
groups: Analysis of variance (F-Tests). 

Variable F df p 

AS13 0.85 5; 146 .52 

0.3S 5i 130 .85 
AS13+14 0.71 5; 130 .62 



As these overall tests of the differences between at! groups are not significant there Is no 
point In carrying out "planned comparisons" for the tests of the factors length" (L) or "density" (D) 
or their interaction (L * D) by rearranging tho groups 2-5 (cf. the design In section 2.1). 

When summarising the results with regard to the achievement scores we have to state: 
The hypothesis that a higher submission density has a positive effect on the achievement is not 
confirmed b> the data. It could not be shown that more frequent opportunities for learner-tutor 
contact (by a higher submission density) result In better achievements. 



4 

Homotaguous non-parametric Kruskal-Waliis H-Tests yield similar results 
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Questionnaire results 

These results wfli be summarized here only; for more details see the project report 
(Hoimberg, Schuemer et si. 1988). 

Course Evaluation 

The evaluation questionnaire contains more than 40 statements of the Ukert type. The 
Subjects (Ss) ware asked to Indicate the degree of agreement on a 4-point scale) and some 
bipolar 7-point scales of the semantic-differential type. 

This questionnaire was sent to "active'' learners who had submitted several assignments and was 
answered by 175 Ss. 

The ratings on the course are on the whole rather positive: Examples of items strongly 
agreed to by the Ss are: 
-04: The material is well structured." 
-011: The manner of presentation is ... informative." 
-Q20: The text made a friendly Impression." 
-Q23: "I felt myself to be challenged by the course to the right degree." 
- 029: 1 liked the manner of presentation on the who!*.* 

In the semantic differential scales the course Is rated as rather "easy, stimulating, clear, 
well structured, appealing, easy to grasp, and motivating". 

A great majority of the Ss expressed satisfaction with the course material as well as the 
correction of and commenting on the assignments (044.1-44.3). 

To simplify the group comparisons and to reduce the great number of variables, a factor 
analysis of the evaluation items (principal factor solution with subsequent VARlMAX-rotation) was 
earned out Three factors were retained and factor scores computed. 
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These factors are: 

EHSSPXJ: The Kerns with high loadings on this factor (a,) have the aspect of 
"stimulation/dullness" In common. Examples of Items with high positive loadings are: 

- Q43.2; The bipolar scale "stimulating/boring" (a^ = .76) 

-09: The statement The course made a rather boring Impression on me" (.73) 

- 043.5: The bipolar scale "appeailng/repeUent (.78) 
Examples of Items with high negative loadings are: 

- 029: The statement T liked the manner of presentation" (-.71) 
-031: The statement The course stimulated me" (-.75) 

- 043.5: The bipolar scale "demotlvating/motlvating" (-.72) 

EacJeUI; The Herns with high loadings on this factor (a„> have the aspect of 
"dfffteulty/slmpllclty- m common. Examples of Items with high positive loadings are: 

- Q18: The statement "I am certain to have understood the essentials" (.63) 

- Q19: The statement 1 was able to study the course rather quickly" (.75) 
Examples of Items with high negative loadings are: 

- 043.1: The bipolar scale "difficult/easy" (a, = .74) 

- 09: The statement The course reaches the limits of my understanding" (.73) 

- 035: The statement "Studying the course was rather strenuous" (.73) 

- 043.7: The bipolar scale "demanding too little /too much" (-.72) 

Factor III '- The Items with high loadings on this factor (a„j) have the aspect of "feelings of 
being addressed personally by the author or the text" 5 in common. Examples of Herns with high 
positive loadings are: 

- 06: The statement The course units made the Impression of a personal letter to me" (.73) 

- Q13: The statement "I have the feeling o' being addressed personally uy the text" (.68) 

- Q25: The statement The manner of presentation made the impression of a personal 

communication between the author and myself" (.68) 
An example of an Item with a high ne g ate ? loading Is: 



This factor refers to Hoimberg's concept of the guided didactic conversation (cf. e.g., Hoimberg, 
Schuemer and Obermeier 1982) 
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- 041: The statement "Being addressed personaBy/the use of personal pronouns (e.g., I and 
you) is an unimportant feature of style (-.64) 



Comparisons erf the experlrrte^ 



Factor scores (standardized: M=0; s-1) were computed and unffactorfal analyses of 
variance were canted out with each of the factor scores respectively as dependent variable and 
with the "groups* as the Independent variable. 



The results are summarised In Table a As can be seen the differences between the 
groups are not significant (for alpha =.01) for any of the factor scores; there Is only a slight 
tendency with regard to factor III: group 1 (with the longest assignments after each course unit) 
have slightly more posttlve scores than the ottw groups (thai means they tend to have a feeling 
of being addressed personally to a greater degree than the other groups; see Table 6). 



Table 6: Comparisons between the groups with regard to course 
evaluation (unlfactorial analyses of variance). 
Independent variable: groups (1-6) 
Dependent variables: factor scores (H»0; s»l) 

a) factor I : boring (+)/ stimulating (-) 

b) factor II : easy (+)/ difficult (-) 

c) factor III: feeling of being addressed personally (+) 



Group Factor scores: 

factor I factor II factor III 







N 


M 


s 


M 


s 


M 


s 




1 


20 


-.07 


0.7 


-.22 


1.2 


.36 


1.1 




2 


36 


-.07 


0.9 


-.07 


1.1 


-.29 


0.9 




3 


32 


.39 


1.4 


-.02 


1.0 


-.04 


1.0 




4 


32 


-.08 


0.8 


-.03 


1.0 


.37 


1.1 




5 


36 


-.01 


1.0 


.18 


0.9 


-.05 


1.0 




6 


19 


-.28 


1.0 


.11 


0.7 


-.28 


0.9 


F 






1.39 




0.51 




2.44 




df 






5; 169 




5; 169 




5? 169 




P 






.23 




.77 




.04 





Obviously, the different versions of assignments or experimental conditions have little 
influence on the overall evaluation of the course by the learners. 
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As the experimental conditions differ mainly in the assignments for course units 1 -12 (I.e., 
their length and their density) the results for some Items referring to the assignments (042. 1 -.7) 
will be presented here in concluding the description of the evaluation results: 

- Tte statement thai the asslg^^ 
bygroupi ^ the targest assignments^ 

- ThM ^assipjm»ms c«mr tew seldom' (042.5) Is more often stated by group 6 (with short 
assignments after every four course units) than in trie other groups 

- Ttat Trie assignir*mt8 are tro 

and 5 (wfth short assignments after each course unit -<33- or after every two course units -G5-) 
to s higher degree than that of the other groups. 

All these tendencies are rather weak, however, and only for Q43.4 ("assignments too 
shorr) significant (Chi 2 -20.2; df-5; p£.0Q1). 

Summarising the evaluation results it can be stated: 

- The ratings on the course are rather positive on the whole 

- The differences between the experimental groups/ conditions wfth regard to the evaluation are 
minimal. 



Non-Starter Questionnaire 

This questionnaire was answered by 107 course participants who had not submitted any 
assignments. 

The questionnaire contains more than 40 statements referring to possible reasons for the 
non-submittai of assignments. The Subjects (Ss) could respond to each statement on a 4-point 
scale: 0, not applicable; 1, little »; 2, largely «; 3, completely applicable. 

Reasons for non-submlttal mentioned most frequently as (more or less) applicable (scale 
values 1-3) are for example: 

- Q10: "personal clrcurratances (such as Alness, family conditions etc.)" 

- Q19- too little time' 
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Reasons referring to the quality of the material or that of the assignments are, on the other hand, 
seldom mentioned. 

The differences between the experimental groups as to the reasons for non-submitting 
are very small. Therefore, they are not presented here in more detail. 

Drop-out Questionnaire 

This questionnaire was answered by 296 course participants who submitted some 
a s signm ents at first but ceased submitting later on 

The questionnaire contains more than 30 statements referring to possible reasons for 
"dropping out". Again the Subjects (Ss) could respond to each statement on a 4-point scale: 0, 
not applicable; 1, little * ; 2, largely «; 3, completely applicable. 

The reasons of the "drop outs" for stopping submitting are similar to those of the non- 
starters. Reasons mentioned most frequently as (more or less) applicable (scale values 1-3) are, 
for example: 

- Q1 : "too little time" 

- Q30d: "Studying the course consumed too much time and work" 

- Q10 : "personal circumstances (such as illness eta)" 

Reasons referring to the quality of the material or that of the assignments are 
unfrequentfy mentioned even by the "drop outs". This pattern of reasons for drop out is also 
known from other courses of the FemUnfversHSt (cf. Schuemer 1979 for a course on 
accountancy). 
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DISCUSSION 

The results provide no basis for confidence in statements about the Impact of various 
frequencies of tutor-student contacts. No statistically significant differences between the groups 
were found with regard to course completion or achievement. There was an unexpected 
tendency, however, in that the groups submitting an assignment after every two course units 
tended to produce more assignment solutions and to continue their course work longer than 
trrase required to suc^ an assignmert 

that more frequent tutor-student contacts lead to greater success (Jowbt drop-out rates). 

Both the submission frequency and the completion rate are lowest hi the group that 
was given the most extensive assignments and was invited to submit an assignment per course 
unit This is remarkable as the course author assumed that this assignment size and 
communication frequency would be most likely to favour the attainment of the course objectives. 
There can thus be no doubt mat, in Popper's terminology, the main hypothesis on which the 
study was based has been falsified. Neither in BaatfVs investigation nor in the one reported \me 
has high assignment frequency ("density in Baath's terminology) been shown to cause higher 
completion rates or better achievement 

Another Z1FF study analysing data from 197 distance-teaching organisations in various 
countries points in the same direction. Most of these organisations offer one assignment per 
course unit. They are q& superior as to higher completion rates or lower non-starter rates in 
comparison with those organisations that offer fewer submission opportunities. 1 



There is no denying that the outcome of these studies is disappointing. While It is no 
longer possible to hypothesize that higher submission density generally favours completion and 
good results ft still seems reasonable to assume that appropriate assignment frequency, 
whatever that may be, exerts favourable Influence. What is appropriate no doubt varies with 
subjects, degrees of difficulty, students' prior knowledge, time available for study and other 
factors. In the present case the course author reefs inclined to draw the conclusion ttat if each 



^^S^L m ** tBPOneti °° in Graff md Holmber9 (1988 > 0l **>™ ' *wed to hare 

» net mentioned 
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unit is to be followed by an assignment for submission, the course unit should be longer (contain 
more learning material) than the units of th9 course iL«d for this study. 

With hindsight It Is possible to see that the present study was not conducted under 
ideal conditions. The design of the research and the way It wee carried out may even have 
prompted the reported outcome. Thus, the students who were expected to solve the most 
extensive assignments (group 1) were given too heavy a workload considering the fact that the 
course was an additional one beside their normal university study and beside renumerative work, 
household duties, social commitments, etc. typical of distance students. The very fact that this 
group was given the most thorough teaching hindered course completion and caused early drop 
out. 

Those students, on the other hand, who were asked to submit the very short 
assignments after each course unit, seem to have found the tasks too bitty and little rewarding 
(Group 3), which could have been foreseen. 

Submitting assignments after two course units at a time may have given more of an 
apparent opportunity to repeat and thus secure what had been learned than after only one; the 
time and work Invested In this may have been fe ieid more In relation to the endeavour 
required and to give enough feedback, particular} <ach course unit was fairly short This 
assumption cannot be said to be corroborated by i» ...-sent study although the tendency 
mentioned above makes ft plausible. 

There was no extrinsic motivation to complete the course as It was not par* of a degree 
programme or any other type of study leading to recognised competence. 

For a possible further replication these difficulties must be avoided and a somewhat 
exact operationalisatlon of the concept of appropriate submission density should be brought 
about 
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Submission Density, Amount of Submission Questions, and 
Quality of Student-Tutor Dialogue - 
A Comment on Holmberg & Schuemer 

John A B&dth, Saitsj&baden 

INTRODUCTION 



In the early seventies, when the experimental studies within an EHSC 1 research project were 
planned, traditional correspondence education was stfil the dominating form of distance 
education. In correspondence education, the postal two-way communication between student 
and tutor was, quite reasonably, considered highly Important (cf. Baath & WangdahJ 1976). 

At the same time, however, there was a clear tendency among correspondence teaching 
institutions to reduce the amount of postal two-way communication In their courses. Above all, 
what I called the "submission density" was getting lower, l.e. the number of submissions were 
gradually reduced. In many cases the total amount of submission questions was also 
substantially diminished. This observation was confirmed by means of an empirical Investigation 
(Baath 1976). It was therefore no wonder that the question came up how this reduction might 
affect the students' study perseverance, their learning results, and their attitudes to the studies. 

This was, briefly speaking, the background of two of the experimental studies within the EHSC 
research protect 

(1) on the submission density of assignments, and 

(2) on assignments for submission being partly replaced by self-checking exercises. 

As is obvious from my preliminary paper (Baath 1979), as well as from my dissertation (Baath 
1980), most of the hypotheses about the effects of varying the submission density had to be 
rejected. It could only be shown that higher submission density entafled a stronger tendency to 
start sending in assignments, and that students with low submission density were more In favour 
of an Increase of submissions than students with high submission density. With regard to the 
second problem, the experiments indicated that ft might be possible to replace at least half of the 
assignment questions by self-checking exercises without any measurable differences in students* 
study perseverance, achievements or attitudes to the study. 



1} EHSC - European Home Study Council, now merged vwth another organisation In the Association of 
European Correspondence Schools (AECS) m 
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THE Z3FF STUDY 

it is, however, stfil quite reasonable to believe that the postal two-way communication is of great 
Importance in distance education of the correspondence education type. It is therefore most 
satisfying that ZIFF has made e new experimental study on submission density and amount of 
submission work. 

Almost no differences at all between groups are discemable in the results from the ZIFF 
experiment. This Is well in agreement with the outcome of the EHSC experiments. May we 
conclude that variables like submission density and amount of submission work are of no 
Importance in distance education of this type? Hardly. It may of course be so, but >t is also 
possible that the (non-) results in both settings are due to other facors> which have not been 
considered or fully controlled in the experimental studies. 

In the discussion part of their report, Holmberg & Schuemer call the reader's attention to a 
number of factors which may have contributed to the lack of significant experimental differences 
in their own study. In addition, I would like to point out two other possible factors: 

(1) the measure of submission density 

(2) the work of the tutors. 

The M6S«'jr<t of Submission Density 

Before varying the submission density of a distance study course, St would appear reasonable to 
establish Its actual, or basic, submission density. How extensive is the amount of work 
demanded by the whole course in its basic form, divided by the original number of submissions? 
When we know that, we are able to make hypotheses about what will happen if we vary the 
number of submissions and, by that, the submission density of the course. 

Holmberg and Schuemer stats the number of pages of the whole course, from which ft is 
possible to calculate the submission density. Number of pages per submission Is a very crude 
measure, however. It would therefore have been valuable to know also the study time needed. 
With an empirically established average net study time - in hours - we have probably a much 
better starting-point for the design of an experimental study on submission density (cf. Baath 
1975). For example, If we find out that - as an average - students spend 96 hours on a course 
which has 8 submissions, we know that the "nornv ' (or baste, or actual) submission density of 
this course Is 12 hours of study per submission. If so, we are in a good position to Judge how the 

so 
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number of submissions can be systematically varied. We may conclude that In a course of this 
type, with Its specific target group, It Is not meaningful to give one of the experiment groups the 
double number of submissions, because the submission density Is already comparatively Mgr.. 
Instead, we may give one experiment group 4 submissions and another 2 submissions - U. 
about 24 and 48 hours of study per submission, respectively. 

Without mis knowledge about study time we do not know enough, I think, about what the 
experimental variations actually mean, It is possible that an empirical establishment of the 
average net study time of the course Essentials of English would have led up to a different 
decision concerning the variation of the submission density man was actually made in the ZIFF 
experiment 

The Work of the Tutors 

The rationale of the experiments on submission density - both the EHSC and the 21FF 
experiments - may be expressed as follows: 

If the work of the tutors In the postal two- way communication Is of great value to the 
students, then It would seem highly probable that the density of postal contacts 
between students and tutors) Is Important. 

It would therefore have been of great Interest to see what this supposedly valuable work of the 
tutors was actually like, and how It was organised. First: There were two tutors. How were they 
assigned to students In the different experiment groups? Second: What was the external turn- 
round time - La how fast did the students get their assignments back? Further What did the 
tutors actually do? What kind of dialogue took place between students and tutor(s)? To what 
extent were the tutors able to take en active Interest in each student and to treat him/her as an 
Individual with specific learning needs and, per'^ps, specific learning problems? What did they 
do in order to encourage the students, to motivate them, to till them with enthusiasm? And finally: 
Did they have any other contacts with their students than on the submission occasions, by mail 
or by telephone? 

The only Information about the tutors and their work, provided In the ZIFF report, Is the following: 

*7ne assignments submitted by the learners were corrected and commented on by 
two teachers of English engaged especial/ for carrying out the project The turn- 
around time for me correction of and commenting on the assignments varied to 
some degree with the workload of the two tutors; but with few exceptions It was 
possible to keep a turn-around time of 10 days or less.' 2 



(Holmberg & Schuemer 1988, p. 8; 1989, p. 56) 




pi 
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On this point the EHSC study was definitely deficient, too. It Is true that my dissertation (Baftth 
'960) contains examples of actual students' answers to assignment questions with tutor's 
corrections and comments, as well as of actual CADE-generated 3 comment letters (pp. 167 - 
173). But a systematical analysis of the tutors's work was never undertaken. 

it is quite possible that with certain kinds of tutor behaviour - for instance, attempts to treat each 
student as an individual and to establish a warm, supporting relationship with each student > 
students wf benefit more from a course arrangement with high submission density than with 
other types of tutor behaviour. With comparatively impersonal tutoring, for instance, it may not 
matter very much - within certain limits - how seldom the students have to submit their 
assignments. 



THE TUTOR AS A FACILITATOR OF LEARNING IN DISTANCE EDUCATION 
The Traditional Postal Two-Way Communication 

The Importance of teacher/tutor behaviours Hke "warmth", empathy and enthusiasm has 
repeatedly been shown in studies of classroom learning (e.g.. Gage 1972, and Brophy & Good 
1974). There Is no reason to believe that distance education differs from face-to-face education in 
this respect, see for example the case reported by Stein (1960) and the findings related to 
Teaming trauma" In distance education reported by Brady (1976). 

lainlng of distance tutors aimed at stimulating such behaviours - preferably in the form of a 
distance study course - could do a lot to create a desirable tutor attitude (or, alternatively, to 
exclude from distance tutoring people who feel that they cannot adopt the appropriate attitude 
for this work). The Association of European Correspondence Schools offers a training course of 
this kind (RekkedaJ 1987), and a lot of separate distance teaching institutions nowadays run their 
own training courses for tutors, trying to inspire them to personalize their tuition, to treat their 
students as Individuals, to encourage them, to make them feel that they have a supporting friend 
in their tutor. A recent training course with this explicit Intention is the one issued by Uber 
Hermods (Baftth 1989). 



3) CADE m ComputeHsssfsted dfatanttMKiucBtfon 



82 



* 



Baath: Comment on Hoimberg « ^uemer / 87 



We must have in mind, however, that the possible Impart of the tutor on his/her students' 
learning, In traditional distance education of the correspondence education type, Is rather limited. 
Normally, the contacts are restricted to the submission occasions, and since the dialogue Is 
♦radWonafly mediated through the post, the communication is most often extremely delayed. A 
student sends In his/her assignment to the school/university. Not untB one, two - or even more - 
weeks later he/she gets the assignment back, with corrections, a grade, and - hopefully - 
constructive and encouraging comments. Usually, the dialogue stops here. Then a new dialogue 
of the same kind starts when the student sends In his/her next assignment (which may very well 
happen before the previous assignment has come back to ti«? student). In a system like this, it Is 
obviously not too easy to act as a genuine "facilitator of learning" (Rogers 1968). 

Nevertheless, the submissions provide an opportunity for contact between student and tutor, for 
real - not simulated - two-way communication at a distance. It was therefore. In 1973, no doubt 
relevant to study the possible effects of varying the submission density (and the amount of 
assignment work), and It still is. 



New Media May Provide Substantially Better Opportunities tor Facilitators of 
Learning in Distance Education 

in the future, however, such research may become less relevant. New media wBI make possible 
other kinds of non-contiguous communication between students and tutor - and between the 
students themselves. Furthermore, this two-way (and multi-way) communication will take place 
not only In relation to the submission of assignments, but much more freely throughout a course. 
Already the telephone is used for such purposes at many distance teaching schools and 
universities, often as a supplement to the two-way communication by mall. With the spreading of 
computers and terminals we are getting a new very powerful medium for almost unlimited 
contiguous two-way and multi-way communication between tutor and students, and between 
students, by meer. of electronic mall and compiler conferencing (e.g., Poulsen & Rekkedal 
1988; NaHu 1989; Mason & Kaye 1989). 

In such a network system of contacts with very short delay, the submission density of a course 
wfli be of less Importance for the contacts between student and tutor. It is also quite probable that 
a distance tutor with the attitude of a genuine facilitator of learning could be of much better help 
to his/her stirlents he^ than In a traditional system where the contacts between tutor and 
student are almost entirely restricted to the correction of and commenting on assignments 
distributed by maO. 
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